- .ipynb_checkpoints
- .localenv
- datasets
- docker
- images
- 01_the_machine_learning_landscape.ipynb
- 02_end_to_end_machine_learning_project.ipynb
- 03_classification.ipynb
- 04_training_linear_models.ipynb
- 05_support_vector_machines.ipynb
- 06_decision_trees.ipynb
- 07_ensemble_learning_and_random_forests.ipynb
- 08_dimensionality_reduction.ipynb
- 09_up_and_running_with_tensorflow.ipynb
- 10_introduction_to_artificial_neural_networks.ipynb
- 11_deep_learning.ipynb
- 12_distributed_tensorflow.ipynb
- 13_convolutional_neural_networks.ipynb
- 14_recurrent_neural_networks.ipynb
- 15_autoencoders.ipynb
- 16_reinforcement_learning.ipynb
- _overview.md
- app_spec.yml
- book_equations.ipynb
- extra_autodiff.ipynb
- extra_capsnets-cn.ipynb
- extra_capsnets.ipynb
- future_encoders.py
- index.ipynb
- LICENSE
- math_linear_algebra.ipynb
- README.md
- requirements.txt
- tensorflow_graph_in_jupyter.py
- tools_matplotlib.ipynb
- tools_numpy.ipynb
- tools_pandas.ipynb
11_deep_learning.ipynb @master — view markup · raw · history · blame
Chapter 11 – Deep Learning
This notebook contains all the sample code and solutions to the exercises in chapter 11.
Setup¶
First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:
In [1]:
# To support both python 2 and python 3
from __future__ import division, print_function, unicode_literals
# Common imports
import numpy as np
import os
# to make this notebook's output stable across runs
def reset_graph(seed=42):
tf.reset_default_graph()
tf.set_random_seed(seed)
np.random.seed(seed)
# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "deep"
def save_fig(fig_id, tight_layout=True):
path = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID, fig_id + ".png")
print("Saving figure", fig_id)
if tight_layout:
plt.tight_layout()
plt.savefig(path, format='png', dpi=300)
Vanishing/Exploding Gradients Problem¶
In [2]:
def logit(z):
return 1 / (1 + np.exp(-z))
In [3]:
z = np.linspace(-5, 5, 200)
plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([-5, 5], [1, 1], 'k--')
plt.plot([0, 0], [-0.2, 1.2], 'k-')
plt.plot([-5, 5], [-3/4, 7/4], 'g--')
plt.plot(z, logit(z), "b-", linewidth=2)
props = dict(facecolor='black', shrink=0.1)
plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha="center")
plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha="center")
plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha="center")
plt.grid(True)
plt.title("Sigmoid activation function", fontsize=14)
plt.axis([-5, 5, -0.2, 1.2])
save_fig("sigmoid_saturation_plot")
plt.show()
Saving figure sigmoid_saturation_plot
Xavier and He Initialization¶
Note: the book uses tensorflow.contrib.layers.fully_connected() rather than tf.layers.dense() (which did not exist when this chapter was written). It is now preferable to use tf.layers.dense(), because anything in the contrib module may change or be deleted without notice. The dense() function is almost identical to the fully_connected() function. The main differences relevant to this chapter are:
- several parameters are renamed:
scopebecomesname,activation_fnbecomesactivation(and similarly the_fnsuffix is removed from other parameters such asnormalizer_fn),weights_initializerbecomeskernel_initializer, etc. - the default
activationis nowNonerather thantf.nn.relu. - it does not support
tensorflow.contrib.framework.arg_scope()(introduced later in chapter 11). - it does not support regularizer params (introduced later in chapter 11).
In [4]:
import tensorflow as tf
In [5]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
In [6]:
he_init = tf.variance_scaling_initializer()
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu,
kernel_initializer=he_init, name="hidden1")
Nonsaturating Activation Functions¶
Leaky ReLU¶
In [7]:
def leaky_relu(z, alpha=0.01):
return np.maximum(alpha*z, z)
In [8]:
plt.plot(z, leaky_relu(z, 0.05), "b-", linewidth=2)
plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([0, 0], [-0.5, 4.2], 'k-')
plt.grid(True)
props = dict(facecolor='black', shrink=0.1)
plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha="center")
plt.title("Leaky ReLU activation function", fontsize=14)
plt.axis([-5, 5, -0.5, 4.2])
save_fig("leaky_relu_plot")
plt.show()
Saving figure leaky_relu_plot
Implementing Leaky ReLU in TensorFlow:
In [9]:
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
In [10]:
def leaky_relu(z, name=None):
return tf.maximum(0.01 * z, z, name=name)
hidden1 = tf.layers.dense(X, n_hidden1, activation=leaky_relu, name="hidden1")
Let's train a neural network on MNIST using the Leaky ReLU. First let's create the graph:
In [11]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
In [12]:
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
In [13]:
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=leaky_relu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=leaky_relu, name="hidden2")
logits = tf.layers.dense(hidden2, n_outputs, name="outputs")
In [14]:
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
In [15]:
learning_rate = 0.01
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
In [16]:
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
In [17]:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
Let's load the data:
Warning: tf.examples.tutorials.mnist is deprecated. We will use tf.keras.datasets.mnist instead.
In [18]:
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0
X_test = X_test.astype(np.float32).reshape(-1, 28*28) / 255.0
y_train = y_train.astype(np.int32)
y_test = y_test.astype(np.int32)
X_valid, X_train = X_train[:5000], X_train[5000:]
y_valid, y_train = y_train[:5000], y_train[5000:]
In [19]:
def shuffle_batch(X, y, batch_size):
rnd_idx = np.random.permutation(len(X))
n_batches = len(X) // batch_size
for batch_idx in np.array_split(rnd_idx, n_batches):
X_batch, y_batch = X[batch_idx], y[batch_idx]
yield X_batch, y_batch
In [20]:
n_epochs = 40
batch_size = 50
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
if epoch % 5 == 0:
acc_batch = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
acc_valid = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Batch accuracy:", acc_batch, "Validation accuracy:", acc_valid)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Batch accuracy: 0.86 Validation accuracy: 0.9048 5 Batch accuracy: 0.94 Validation accuracy: 0.9494 10 Batch accuracy: 0.92 Validation accuracy: 0.9656 15 Batch accuracy: 0.94 Validation accuracy: 0.9712 20 Batch accuracy: 1.0 Validation accuracy: 0.9764 25 Batch accuracy: 1.0 Validation accuracy: 0.9772 30 Batch accuracy: 0.98 Validation accuracy: 0.9782 35 Batch accuracy: 1.0 Validation accuracy: 0.9786
ELU¶
In [21]:
def elu(z, alpha=1):
return np.where(z < 0, alpha * (np.exp(z) - 1), z)
In [22]:
plt.plot(z, elu(z), "b-", linewidth=2)
plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([-5, 5], [-1, -1], 'k--')
plt.plot([0, 0], [-2.2, 3.2], 'k-')
plt.grid(True)
plt.title(r"ELU activation function ($\alpha=1$)", fontsize=14)
plt.axis([-5, 5, -2.2, 3.2])
save_fig("elu_plot")
plt.show()
Saving figure elu_plot
Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:
In [23]:
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
In [24]:
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.elu, name="hidden1")
SELU¶
This activation function was proposed in this great paper by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017 (I will definitely add it to the book). During training, a neural network composed of a stack of dense layers using the SELU activation function will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out.
In [25]:
def selu(z,
scale=1.0507009873554804934193349852946,
alpha=1.6732632423543772848170429916717):
return scale * elu(z, alpha)
In [26]:
plt.plot(z, selu(z), "b-", linewidth=2)
plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([-5, 5], [-1.758, -1.758], 'k--')
plt.plot([0, 0], [-2.2, 3.2], 'k-')
plt.grid(True)
plt.title(r"SELU activation function", fontsize=14)
plt.axis([-5, 5, -2.2, 3.2])
save_fig("selu_plot")
plt.show()
Saving figure selu_plot
By default, the SELU hyperparameters (scale and alpha) are tuned in such a way that the mean remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 100 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:
In [27]:
np.random.seed(42)
Z = np.random.normal(size=(500, 100))
for layer in range(100):
W = np.random.normal(size=(100, 100), scale=np.sqrt(1/100))
Z = selu(np.dot(Z, W))
means = np.mean(Z, axis=1)
stds = np.std(Z, axis=1)
if layer % 10 == 0:
print("Layer {}: {:.2f} < mean < {:.2f}, {:.2f} < std deviation < {:.2f}".format(
layer, means.min(), means.max(), stds.min(), stds.max()))
Layer 0: -0.26 < mean < 0.27, 0.74 < std deviation < 1.27 Layer 10: -0.24 < mean < 0.27, 0.74 < std deviation < 1.27 Layer 20: -0.17 < mean < 0.18, 0.74 < std deviation < 1.24 Layer 30: -0.27 < mean < 0.24, 0.78 < std deviation < 1.20 Layer 40: -0.38 < mean < 0.39, 0.74 < std deviation < 1.25 Layer 50: -0.27 < mean < 0.31, 0.73 < std deviation < 1.27 Layer 60: -0.26 < mean < 0.43, 0.74 < std deviation < 1.35 Layer 70: -0.19 < mean < 0.21, 0.75 < std deviation < 1.21 Layer 80: -0.18 < mean < 0.16, 0.72 < std deviation < 1.19 Layer 90: -0.19 < mean < 0.16, 0.75 < std deviation < 1.20
The tf.nn.selu() function was added in TensorFlow 1.4. For earlier versions, you can use the following implementation:
In [28]:
def selu(z,
scale=1.0507009873554804934193349852946,
alpha=1.6732632423543772848170429916717):
return scale * tf.where(z >= 0.0, z, alpha * tf.nn.elu(z))
However, the SELU activation function cannot be used along with regular Dropout (this would cancel the SELU activation function's self-normalizing property). Fortunately, there is a Dropout variant called Alpha Dropout proposed in the same paper. It is available in tf.contrib.nn.alpha_dropout() since TF 1.4 (or check out this implementation by the Institute of Bioinformatics, Johannes Kepler University Linz).
Let's create a neural net for MNIST using the SELU activation function:
In [29]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=selu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=selu, name="hidden2")
logits = tf.layers.dense(hidden2, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
learning_rate = 0.01
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
saver = tf.train.Saver()
n_epochs = 40
batch_size = 50
Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:
In [30]:
means = X_train.mean(axis=0, keepdims=True)
stds = X_train.std(axis=0, keepdims=True) + 1e-10
X_val_scaled = (X_valid - means) / stds
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
X_batch_scaled = (X_batch - means) / stds
sess.run(training_op, feed_dict={X: X_batch_scaled, y: y_batch})
if epoch % 5 == 0:
acc_batch = accuracy.eval(feed_dict={X: X_batch_scaled, y: y_batch})
acc_valid = accuracy.eval(feed_dict={X: X_val_scaled, y: y_valid})
print(epoch, "Batch accuracy:", acc_batch, "Validation accuracy:", acc_valid)
save_path = saver.save(sess, "./my_model_final_selu.ckpt")
0 Batch accuracy: 0.88 Validation accuracy: 0.9232 5 Batch accuracy: 0.98 Validation accuracy: 0.9576 10 Batch accuracy: 1.0 Validation accuracy: 0.9662 15 Batch accuracy: 0.96 Validation accuracy: 0.968 20 Batch accuracy: 1.0 Validation accuracy: 0.9694 25 Batch accuracy: 1.0 Validation accuracy: 0.9688 30 Batch accuracy: 1.0 Validation accuracy: 0.969 35 Batch accuracy: 1.0 Validation accuracy: 0.9696
Batch Normalization¶
Note: the book uses tensorflow.contrib.layers.batch_norm() rather than tf.layers.batch_normalization() (which did not exist when this chapter was written). It is now preferable to use tf.layers.batch_normalization(), because anything in the contrib module may change or be deleted without notice. Instead of using the batch_norm() function as a regularizer parameter to the fully_connected() function, we now use batch_normalization() and we explicitly create a distinct layer. The parameters are a bit different, in particular:
decayis renamed tomomentum,is_trainingis renamed totraining,updates_collectionsis removed: the update operations needed by batch normalization are added to theUPDATE_OPScollection and you need to explicity run these operations during training (see the execution phase below),- we don't need to specify
scale=True, as that is the default.
Also note that in order to run batch norm just before each hidden layer's activation function, we apply the ELU activation function manually, right after the batch norm layer.
Note: since the tf.layers.dense() function is incompatible with tf.contrib.layers.arg_scope() (which is used in the book), we now use python's functools.partial() function instead. It makes it easy to create a my_dense_layer() function that just calls tf.layers.dense() with the desired parameters automatically set (unless they are overridden when calling my_dense_layer()). As you can see, the code remains very similar.
In [31]:
reset_graph()
import tensorflow as tf
n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
training = tf.placeholder_with_default(False, shape=(), name='training')
hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1")
bn1 = tf.layers.batch_normalization(hidden1, training=training, momentum=0.9)
bn1_act = tf.nn.elu(bn1)
hidden2 = tf.layers.dense(bn1_act, n_hidden2, name="hidden2")
bn2 = tf.layers.batch_normalization(hidden2, training=training, momentum=0.9)
bn2_act = tf.nn.elu(bn2)
logits_before_bn = tf.layers.dense(bn2_act, n_outputs, name="outputs")
logits = tf.layers.batch_normalization(logits_before_bn, training=training,
momentum=0.9)
In [32]:
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
training = tf.placeholder_with_default(False, shape=(), name='training')
To avoid repeating the same parameters over and over again, we can use Python's partial() function:
In [33]:
from functools import partial
my_batch_norm_layer = partial(tf.layers.batch_normalization,
training=training, momentum=0.9)
hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1")
bn1 = my_batch_norm_layer(hidden1)
bn1_act = tf.nn.elu(bn1)
hidden2 = tf.layers.dense(bn1_act, n_hidden2, name="hidden2")
bn2 = my_batch_norm_layer(hidden2)
bn2_act = tf.nn.elu(bn2)
logits_before_bn = tf.layers.dense(bn2_act, n_outputs, name="outputs")
logits = my_batch_norm_layer(logits_before_bn)
Let's build a neural net for MNIST, using the ELU activation function and Batch Normalization at each layer:
In [34]:
reset_graph()
batch_norm_momentum = 0.9
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
training = tf.placeholder_with_default(False, shape=(), name='training')
with tf.name_scope("dnn"):
he_init = tf.variance_scaling_initializer()
my_batch_norm_layer = partial(
tf.layers.batch_normalization,
training=training,
momentum=batch_norm_momentum)
my_dense_layer = partial(
tf.layers.dense,
kernel_initializer=he_init)
hidden1 = my_dense_layer(X, n_hidden1, name="hidden1")
bn1 = tf.nn.elu(my_batch_norm_layer(hidden1))
hidden2 = my_dense_layer(bn1, n_hidden2, name="hidden2")
bn2 = tf.nn.elu(my_batch_norm_layer(hidden2))
logits_before_bn = my_dense_layer(bn2, n_outputs, name="outputs")
logits = my_batch_norm_layer(logits_before_bn)
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
saver = tf.train.Saver()
Note: since we are using tf.layers.batch_normalization() rather than tf.contrib.layers.batch_norm() (as in the book), we need to explicitly run the extra update operations needed by batch normalization (sess.run([training_op, extra_update_ops],...).
In [35]:
n_epochs = 20
batch_size = 200
In [36]:
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run([training_op, extra_update_ops],
feed_dict={training: True, X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.9042 1 Validation accuracy: 0.928 2 Validation accuracy: 0.9374 3 Validation accuracy: 0.9474 4 Validation accuracy: 0.9532 5 Validation accuracy: 0.9572 6 Validation accuracy: 0.9626 7 Validation accuracy: 0.9628 8 Validation accuracy: 0.9664 9 Validation accuracy: 0.968 10 Validation accuracy: 0.9694 11 Validation accuracy: 0.9696 12 Validation accuracy: 0.971 13 Validation accuracy: 0.971 14 Validation accuracy: 0.9728 15 Validation accuracy: 0.9734 16 Validation accuracy: 0.9728 17 Validation accuracy: 0.975 18 Validation accuracy: 0.9752 19 Validation accuracy: 0.976
What!? That's not a great accuracy for MNIST. Of course, if you train for longer it will get much better accuracy, but with such a shallow network, Batch Norm and ELU are unlikely to have very positive impact: they shine mostly for much deeper nets.
Note that you could also make the training operation depend on the update operations:
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
training_op = optimizer.minimize(loss)
This way, you would just have to evaluate the training_op during training, TensorFlow would automatically run the update operations as well:
sess.run(training_op, feed_dict={training: True, X: X_batch, y: y_batch})
One more thing: notice that the list of trainable variables is shorter than the list of all global variables. This is because the moving averages are non-trainable variables. If you want to reuse a pretrained neural network (see below), you must not forget these non-trainable variables.
In [37]:
[v.name for v in tf.trainable_variables()]
Out[37]:
['hidden1/kernel:0', 'hidden1/bias:0', 'batch_normalization/gamma:0', 'batch_normalization/beta:0', 'hidden2/kernel:0', 'hidden2/bias:0', 'batch_normalization_1/gamma:0', 'batch_normalization_1/beta:0', 'outputs/kernel:0', 'outputs/bias:0', 'batch_normalization_2/gamma:0', 'batch_normalization_2/beta:0']
In [38]:
[v.name for v in tf.global_variables()]
Out[38]:
['hidden1/kernel:0', 'hidden1/bias:0', 'batch_normalization/gamma:0', 'batch_normalization/beta:0', 'batch_normalization/moving_mean:0', 'batch_normalization/moving_variance:0', 'hidden2/kernel:0', 'hidden2/bias:0', 'batch_normalization_1/gamma:0', 'batch_normalization_1/beta:0', 'batch_normalization_1/moving_mean:0', 'batch_normalization_1/moving_variance:0', 'outputs/kernel:0', 'outputs/bias:0', 'batch_normalization_2/gamma:0', 'batch_normalization_2/beta:0', 'batch_normalization_2/moving_mean:0', 'batch_normalization_2/moving_variance:0']
Gradient Clipping¶
Let's create a simple neural net for MNIST and add gradient clipping. The first part is the same as earlier (except we added a few more layers to demonstrate reusing pretrained models, see below):
In [39]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 50
n_hidden3 = 50
n_hidden4 = 50
n_hidden5 = 50
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2")
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, name="hidden3")
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name="hidden4")
hidden5 = tf.layers.dense(hidden4, n_hidden5, activation=tf.nn.relu, name="hidden5")
logits = tf.layers.dense(hidden5, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
In [40]:
learning_rate = 0.01
Now we apply gradient clipping. For this, we need to get the gradients, use the clip_by_value() function to clip them, then apply them:
In [41]:
threshold = 1.0
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
grads_and_vars = optimizer.compute_gradients(loss)
capped_gvs = [(tf.clip_by_value(grad, -threshold, threshold), var)
for grad, var in grads_and_vars]
training_op = optimizer.apply_gradients(capped_gvs)
The rest is the same as usual:
In [42]:
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
In [43]:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [44]:
n_epochs = 20
batch_size = 200
In [45]:
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.2876 1 Validation accuracy: 0.7944 2 Validation accuracy: 0.8796 3 Validation accuracy: 0.906 4 Validation accuracy: 0.9162 5 Validation accuracy: 0.9216 6 Validation accuracy: 0.9292 7 Validation accuracy: 0.9356 8 Validation accuracy: 0.938 9 Validation accuracy: 0.9416 10 Validation accuracy: 0.9458 11 Validation accuracy: 0.947 12 Validation accuracy: 0.9476 13 Validation accuracy: 0.9534 14 Validation accuracy: 0.9562 15 Validation accuracy: 0.9566 16 Validation accuracy: 0.9576 17 Validation accuracy: 0.9588 18 Validation accuracy: 0.9624 19 Validation accuracy: 0.961
Reusing Pretrained Layers¶
Reusing a TensorFlow Model¶
First you need to load the graph's structure. The import_meta_graph() function does just that, loading the graph's operations into the default graph, and returning a Saver that you can then use to restore the model's state. Note that by default, a Saver saves the structure of the graph into a .meta file, so that's the file you should load:
In [46]:
reset_graph()
In [47]:
saver = tf.train.import_meta_graph("./my_model_final.ckpt.meta")
Next you need to get a handle on all the operations you will need for training. If you don't know the graph's structure, you can list all the operations:
In [48]:
for op in tf.get_default_graph().get_operations():
print(op.name)
save/RestoreV2/shape_and_slices save/RestoreV2/tensor_names save/SaveV2/shape_and_slices save/SaveV2/tensor_names save/Const save/RestoreV2 eval/Const eval/in_top_k/InTopKV2/k GradientDescent/learning_rate clip_by_value_11/clip_value_max clip_by_value_11/clip_value_min clip_by_value_10/clip_value_max clip_by_value_10/clip_value_min clip_by_value_9/clip_value_max clip_by_value_9/clip_value_min clip_by_value_8/clip_value_max clip_by_value_8/clip_value_min clip_by_value_7/clip_value_max clip_by_value_7/clip_value_min clip_by_value_6/clip_value_max clip_by_value_6/clip_value_min clip_by_value_5/clip_value_max clip_by_value_5/clip_value_min clip_by_value_4/clip_value_max clip_by_value_4/clip_value_min clip_by_value_3/clip_value_max clip_by_value_3/clip_value_min clip_by_value_2/clip_value_max clip_by_value_2/clip_value_min clip_by_value_1/clip_value_max clip_by_value_1/clip_value_min clip_by_value/clip_value_max clip_by_value/clip_value_min gradients/loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits_grad/ExpandDims/dim gradients/loss/loss_grad/Maximum/y gradients/loss/loss_grad/Const_1 gradients/loss/loss_grad/Const gradients/loss/loss_grad/Shape_2 gradients/loss/loss_grad/Prod_1 gradients/loss/loss_grad/Maximum gradients/loss/loss_grad/Reshape/shape gradients/grad_ys_0 gradients/Shape gradients/Fill gradients/loss/loss_grad/Reshape loss/Const outputs/bias save/Assign_10 outputs/bias/read outputs/bias/Initializer/zeros outputs/bias/Assign outputs/kernel save/Assign_11 outputs/kernel/read outputs/kernel/Initializer/random_uniform/max outputs/kernel/Initializer/random_uniform/min outputs/kernel/Initializer/random_uniform/sub outputs/kernel/Initializer/random_uniform/shape outputs/kernel/Initializer/random_uniform/RandomUniform outputs/kernel/Initializer/random_uniform/mul outputs/kernel/Initializer/random_uniform outputs/kernel/Assign hidden5/bias save/Assign_8 hidden5/bias/read hidden5/bias/Initializer/zeros hidden5/bias/Assign hidden5/kernel save/Assign_9 hidden5/kernel/read hidden5/kernel/Initializer/random_uniform/max hidden5/kernel/Initializer/random_uniform/min hidden5/kernel/Initializer/random_uniform/sub hidden5/kernel/Initializer/random_uniform/shape hidden5/kernel/Initializer/random_uniform/RandomUniform hidden5/kernel/Initializer/random_uniform/mul hidden5/kernel/Initializer/random_uniform hidden5/kernel/Assign hidden4/bias save/Assign_6 hidden4/bias/read hidden4/bias/Initializer/zeros hidden4/bias/Assign hidden4/kernel save/Assign_7 hidden4/kernel/read hidden4/kernel/Initializer/random_uniform/max hidden4/kernel/Initializer/random_uniform/min hidden4/kernel/Initializer/random_uniform/sub hidden4/kernel/Initializer/random_uniform/shape hidden4/kernel/Initializer/random_uniform/RandomUniform hidden4/kernel/Initializer/random_uniform/mul hidden4/kernel/Initializer/random_uniform hidden4/kernel/Assign hidden3/bias save/Assign_4 hidden3/bias/read hidden3/bias/Initializer/zeros hidden3/bias/Assign hidden3/kernel save/Assign_5 hidden3/kernel/read hidden3/kernel/Initializer/random_uniform/max hidden3/kernel/Initializer/random_uniform/min hidden3/kernel/Initializer/random_uniform/sub hidden3/kernel/Initializer/random_uniform/shape hidden3/kernel/Initializer/random_uniform/RandomUniform hidden3/kernel/Initializer/random_uniform/mul hidden3/kernel/Initializer/random_uniform hidden3/kernel/Assign hidden2/bias save/Assign_2 hidden2/bias/read hidden2/bias/Initializer/zeros hidden2/bias/Assign hidden2/kernel save/Assign_3 hidden2/kernel/read hidden2/kernel/Initializer/random_uniform/max hidden2/kernel/Initializer/random_uniform/min hidden2/kernel/Initializer/random_uniform/sub hidden2/kernel/Initializer/random_uniform/shape hidden2/kernel/Initializer/random_uniform/RandomUniform hidden2/kernel/Initializer/random_uniform/mul hidden2/kernel/Initializer/random_uniform hidden2/kernel/Assign hidden1/bias save/Assign hidden1/bias/read hidden1/bias/Initializer/zeros hidden1/bias/Assign hidden1/kernel save/Assign_1 save/restore_all save/SaveV2 save/control_dependency hidden1/kernel/read hidden1/kernel/Initializer/random_uniform/max hidden1/kernel/Initializer/random_uniform/min hidden1/kernel/Initializer/random_uniform/sub hidden1/kernel/Initializer/random_uniform/shape hidden1/kernel/Initializer/random_uniform/RandomUniform hidden1/kernel/Initializer/random_uniform/mul hidden1/kernel/Initializer/random_uniform hidden1/kernel/Assign init y loss/SparseSoftmaxCrossEntropyWithLogits/Shape X dnn/hidden1/MatMul dnn/hidden1/BiasAdd dnn/hidden1/Relu dnn/hidden2/MatMul dnn/hidden2/BiasAdd dnn/hidden2/Relu dnn/hidden3/MatMul dnn/hidden3/BiasAdd dnn/hidden3/Relu dnn/hidden4/MatMul dnn/hidden4/BiasAdd dnn/hidden4/Relu dnn/hidden5/MatMul dnn/hidden5/BiasAdd dnn/hidden5/Relu dnn/outputs/MatMul dnn/outputs/BiasAdd eval/in_top_k/InTopKV2 eval/Cast eval/accuracy loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits gradients/loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits_grad/PreventGradient gradients/zeros_like gradients/loss/loss_grad/Shape_1 gradients/loss/loss_grad/Prod gradients/loss/loss_grad/floordiv gradients/loss/loss_grad/Cast gradients/loss/loss_grad/Shape gradients/loss/loss_grad/Tile gradients/loss/loss_grad/truediv gradients/loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits_grad/ExpandDims gradients/loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits_grad/mul gradients/dnn/outputs/BiasAdd_grad/BiasAddGrad gradients/dnn/outputs/BiasAdd_grad/tuple/group_deps gradients/dnn/outputs/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_11 GradientDescent/update_outputs/bias/ApplyGradientDescent gradients/dnn/outputs/BiasAdd_grad/tuple/control_dependency gradients/dnn/outputs/MatMul_grad/MatMul_1 gradients/dnn/outputs/MatMul_grad/MatMul gradients/dnn/outputs/MatMul_grad/tuple/group_deps gradients/dnn/outputs/MatMul_grad/tuple/control_dependency_1 clip_by_value_10 GradientDescent/update_outputs/kernel/ApplyGradientDescent gradients/dnn/outputs/MatMul_grad/tuple/control_dependency gradients/dnn/hidden5/Relu_grad/ReluGrad gradients/dnn/hidden5/BiasAdd_grad/BiasAddGrad gradients/dnn/hidden5/BiasAdd_grad/tuple/group_deps gradients/dnn/hidden5/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_9 GradientDescent/update_hidden5/bias/ApplyGradientDescent gradients/dnn/hidden5/BiasAdd_grad/tuple/control_dependency gradients/dnn/hidden5/MatMul_grad/MatMul_1 gradients/dnn/hidden5/MatMul_grad/MatMul gradients/dnn/hidden5/MatMul_grad/tuple/group_deps gradients/dnn/hidden5/MatMul_grad/tuple/control_dependency_1 clip_by_value_8 GradientDescent/update_hidden5/kernel/ApplyGradientDescent gradients/dnn/hidden5/MatMul_grad/tuple/control_dependency gradients/dnn/hidden4/Relu_grad/ReluGrad gradients/dnn/hidden4/BiasAdd_grad/BiasAddGrad gradients/dnn/hidden4/BiasAdd_grad/tuple/group_deps gradients/dnn/hidden4/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_7 GradientDescent/update_hidden4/bias/ApplyGradientDescent gradients/dnn/hidden4/BiasAdd_grad/tuple/control_dependency gradients/dnn/hidden4/MatMul_grad/MatMul_1 gradients/dnn/hidden4/MatMul_grad/MatMul gradients/dnn/hidden4/MatMul_grad/tuple/group_deps gradients/dnn/hidden4/MatMul_grad/tuple/control_dependency_1 clip_by_value_6 GradientDescent/update_hidden4/kernel/ApplyGradientDescent gradients/dnn/hidden4/MatMul_grad/tuple/control_dependency gradients/dnn/hidden3/Relu_grad/ReluGrad gradients/dnn/hidden3/BiasAdd_grad/BiasAddGrad gradients/dnn/hidden3/BiasAdd_grad/tuple/group_deps gradients/dnn/hidden3/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_5 GradientDescent/update_hidden3/bias/ApplyGradientDescent gradients/dnn/hidden3/BiasAdd_grad/tuple/control_dependency gradients/dnn/hidden3/MatMul_grad/MatMul_1 gradients/dnn/hidden3/MatMul_grad/MatMul gradients/dnn/hidden3/MatMul_grad/tuple/group_deps gradients/dnn/hidden3/MatMul_grad/tuple/control_dependency_1 clip_by_value_4 GradientDescent/update_hidden3/kernel/ApplyGradientDescent gradients/dnn/hidden3/MatMul_grad/tuple/control_dependency gradients/dnn/hidden2/Relu_grad/ReluGrad gradients/dnn/hidden2/BiasAdd_grad/BiasAddGrad gradients/dnn/hidden2/BiasAdd_grad/tuple/group_deps gradients/dnn/hidden2/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_3 GradientDescent/update_hidden2/bias/ApplyGradientDescent gradients/dnn/hidden2/BiasAdd_grad/tuple/control_dependency gradients/dnn/hidden2/MatMul_grad/MatMul_1 gradients/dnn/hidden2/MatMul_grad/MatMul gradients/dnn/hidden2/MatMul_grad/tuple/group_deps gradients/dnn/hidden2/MatMul_grad/tuple/control_dependency_1 clip_by_value_2 GradientDescent/update_hidden2/kernel/ApplyGradientDescent gradients/dnn/hidden2/MatMul_grad/tuple/control_dependency gradients/dnn/hidden1/Relu_grad/ReluGrad gradients/dnn/hidden1/BiasAdd_grad/BiasAddGrad gradients/dnn/hidden1/BiasAdd_grad/tuple/group_deps gradients/dnn/hidden1/BiasAdd_grad/tuple/control_dependency_1 clip_by_value_1 GradientDescent/update_hidden1/bias/ApplyGradientDescent gradients/dnn/hidden1/BiasAdd_grad/tuple/control_dependency gradients/dnn/hidden1/MatMul_grad/MatMul_1 gradients/dnn/hidden1/MatMul_grad/MatMul gradients/dnn/hidden1/MatMul_grad/tuple/group_deps gradients/dnn/hidden1/MatMul_grad/tuple/control_dependency_1 clip_by_value GradientDescent/update_hidden1/kernel/ApplyGradientDescent GradientDescent gradients/dnn/hidden1/MatMul_grad/tuple/control_dependency loss/loss
Oops, that's a lot of operations! It's much easier to use TensorBoard to visualize the graph. The following hack will allow you to visualize the graph within Jupyter (if it does not work with your browser, you will need to use a FileWriter to save the graph and then visualize it in TensorBoard):
In [49]:
from tensorflow_graph_in_jupyter import show_graph
In [50]:
show_graph(tf.get_default_graph())
Once you know which operations you need, you can get a handle on them using the graph's get_operation_by_name() or get_tensor_by_name() methods:
In [51]:
X = tf.get_default_graph().get_tensor_by_name("X:0")
y = tf.get_default_graph().get_tensor_by_name("y:0")
accuracy = tf.get_default_graph().get_tensor_by_name("eval/accuracy:0")
training_op = tf.get_default_graph().get_operation_by_name("GradientDescent")
If you are the author of the original model, you could make things easier for people who will reuse your model by giving operations very clear names and documenting them. Another approach is to create a collection containing all the important operations that people will want to get a handle on:
In [52]:
for op in (X, y, accuracy, training_op):
tf.add_to_collection("my_important_ops", op)
This way people who reuse your model will be able to simply write:
In [53]:
X, y, accuracy, training_op = tf.get_collection("my_important_ops")
Now you can start a session, restore the model's state and continue training on your data:
In [54]:
with tf.Session() as sess:
saver.restore(sess, "./my_model_final.ckpt")
# continue training the model...
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt
Actually, let's test this for real!
In [55]:
with tf.Session() as sess:
saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.9644 1 Validation accuracy: 0.9646 2 Validation accuracy: 0.9624 3 Validation accuracy: 0.9658 4 Validation accuracy: 0.9668 5 Validation accuracy: 0.9652 6 Validation accuracy: 0.9678 7 Validation accuracy: 0.9676 8 Validation accuracy: 0.9676 9 Validation accuracy: 0.969 10 Validation accuracy: 0.97 11 Validation accuracy: 0.9706 12 Validation accuracy: 0.9676 13 Validation accuracy: 0.9688 14 Validation accuracy: 0.9708 15 Validation accuracy: 0.9696 16 Validation accuracy: 0.9712 17 Validation accuracy: 0.9696 18 Validation accuracy: 0.9688 19 Validation accuracy: 0.9724
Alternatively, if you have access to the Python code that built the original graph, you can use it instead of import_meta_graph():
In [56]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 50
n_hidden3 = 50
n_hidden4 = 50
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2")
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, name="hidden3")
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name="hidden4")
hidden5 = tf.layers.dense(hidden4, n_hidden5, activation=tf.nn.relu, name="hidden5")
logits = tf.layers.dense(hidden5, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
learning_rate = 0.01
threshold = 1.0
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
grads_and_vars = optimizer.compute_gradients(loss)
capped_gvs = [(tf.clip_by_value(grad, -threshold, threshold), var)
for grad, var in grads_and_vars]
training_op = optimizer.apply_gradients(capped_gvs)
init = tf.global_variables_initializer()
saver = tf.train.Saver()
And continue training:
In [57]:
with tf.Session() as sess:
saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.964 1 Validation accuracy: 0.9644 2 Validation accuracy: 0.9626 3 Validation accuracy: 0.965 4 Validation accuracy: 0.9668 5 Validation accuracy: 0.9656 6 Validation accuracy: 0.968 7 Validation accuracy: 0.9678 8 Validation accuracy: 0.9676 9 Validation accuracy: 0.9686 10 Validation accuracy: 0.9698 11 Validation accuracy: 0.9708 12 Validation accuracy: 0.9678 13 Validation accuracy: 0.9688 14 Validation accuracy: 0.9708 15 Validation accuracy: 0.9694 16 Validation accuracy: 0.9714 17 Validation accuracy: 0.9696 18 Validation accuracy: 0.969 19 Validation accuracy: 0.9724
In general you will want to reuse only the lower layers. If you are using import_meta_graph() it will load the whole graph, but you can simply ignore the parts you do not need. In this example, we add a new 4th hidden layer on top of the pretrained 3rd layer (ignoring the old 4th hidden layer). We also build a new output layer, the loss for this new output, and a new optimizer to minimize it. We also need another saver to save the whole graph (containing both the entire old graph plus the new operations), and an initialization operation to initialize all the new variables:
In [58]:
reset_graph()
n_hidden4 = 20 # new layer
n_outputs = 10 # new layer
saver = tf.train.import_meta_graph("./my_model_final.ckpt.meta")
X = tf.get_default_graph().get_tensor_by_name("X:0")
y = tf.get_default_graph().get_tensor_by_name("y:0")
hidden3 = tf.get_default_graph().get_tensor_by_name("dnn/hidden4/Relu:0")
new_hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name="new_hidden4")
new_logits = tf.layers.dense(new_hidden4, n_outputs, name="new_outputs")
with tf.name_scope("new_loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=new_logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("new_eval"):
correct = tf.nn.in_top_k(new_logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
with tf.name_scope("new_train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
new_saver = tf.train.Saver()
And we can train this new model:
In [59]:
with tf.Session() as sess:
init.run()
saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = new_saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.924 1 Validation accuracy: 0.9424 2 Validation accuracy: 0.9512 3 Validation accuracy: 0.9562 4 Validation accuracy: 0.9606 5 Validation accuracy: 0.9568 6 Validation accuracy: 0.9622 7 Validation accuracy: 0.9638 8 Validation accuracy: 0.9632 9 Validation accuracy: 0.9644 10 Validation accuracy: 0.966 11 Validation accuracy: 0.9666 12 Validation accuracy: 0.9636 13 Validation accuracy: 0.9676 14 Validation accuracy: 0.9694 15 Validation accuracy: 0.968 16 Validation accuracy: 0.9692 17 Validation accuracy: 0.968 18 Validation accuracy: 0.9698 19 Validation accuracy: 0.9692
If you have access to the Python code that built the original graph, you can just reuse the parts you need and drop the rest:
In [60]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300 # reused
n_hidden2 = 50 # reused
n_hidden3 = 50 # reused
n_hidden4 = 20 # new!
n_outputs = 10 # new!
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1") # reused
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2") # reused
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, name="hidden3") # reused
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name="hidden4") # new!
logits = tf.layers.dense(hidden4, n_outputs, name="outputs") # new!
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
However, you must create one Saver to restore the pretrained model (giving it the list of variables to restore, or else it will complain that the graphs don't match), and another Saver to save the new model, once it is trained:
In [61]:
reuse_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="hidden[123]") # regular expression
restore_saver = tf.train.Saver(reuse_vars) # to restore layers 1-3
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs): # not shown in the book
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size): # not shown
sess.run(training_op, feed_dict={X: X_batch, y: y_batch}) # not shown
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid}) # not shown
print(epoch, "Validation accuracy:", accuracy_val) # not shown
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.9132 1 Validation accuracy: 0.9318 2 Validation accuracy: 0.9442 3 Validation accuracy: 0.95 4 Validation accuracy: 0.9524 5 Validation accuracy: 0.9544 6 Validation accuracy: 0.9558 7 Validation accuracy: 0.9606 8 Validation accuracy: 0.9616 9 Validation accuracy: 0.9604 10 Validation accuracy: 0.962 11 Validation accuracy: 0.9648 12 Validation accuracy: 0.9604 13 Validation accuracy: 0.968 14 Validation accuracy: 0.9664 15 Validation accuracy: 0.9676 16 Validation accuracy: 0.9692 17 Validation accuracy: 0.9694 18 Validation accuracy: 0.969 19 Validation accuracy: 0.9674
Reusing Models from Other Frameworks¶
In this example, for each variable we want to reuse, we find its initializer's assignment operation, and we get its second input, which corresponds to the initialization value. When we run the initializer, we replace the initialization values with the ones we want, using a feed_dict:
In [62]:
reset_graph()
n_inputs = 2
n_hidden1 = 3
In [63]:
original_w = [[1., 2., 3.], [4., 5., 6.]] # Load the weights from the other framework
original_b = [7., 8., 9.] # Load the biases from the other framework
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
# [...] Build the rest of the model
# Get a handle on the assignment nodes for the hidden1 variables
graph = tf.get_default_graph()
assign_kernel = graph.get_operation_by_name("hidden1/kernel/Assign")
assign_bias = graph.get_operation_by_name("hidden1/bias/Assign")
init_kernel = assign_kernel.inputs[1]
init_bias = assign_bias.inputs[1]
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init, feed_dict={init_kernel: original_w, init_bias: original_b})
# [...] Train the model on your new task
print(hidden1.eval(feed_dict={X: [[10.0, 11.0]]})) # not shown in the book
[[ 61. 83. 105.]]
Note: the weights variable created by the tf.layers.dense() function is called "kernel" (instead of "weights" when using the tf.contrib.layers.fully_connected(), as in the book), and the biases variable is called bias instead of biases.
Another approach (initially used in the book) would be to create dedicated assignment nodes and dedicated placeholders. This is more verbose and less efficient, but you may find this more explicit:
In [64]:
reset_graph()
n_inputs = 2
n_hidden1 = 3
original_w = [[1., 2., 3.], [4., 5., 6.]] # Load the weights from the other framework
original_b = [7., 8., 9.] # Load the biases from the other framework
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
# [...] Build the rest of the model
# Get a handle on the variables of layer hidden1
with tf.variable_scope("", default_name="", reuse=True): # root scope
hidden1_weights = tf.get_variable("hidden1/kernel")
hidden1_biases = tf.get_variable("hidden1/bias")
# Create dedicated placeholders and assignment nodes
original_weights = tf.placeholder(tf.float32, shape=(n_inputs, n_hidden1))
original_biases = tf.placeholder(tf.float32, shape=n_hidden1)
assign_hidden1_weights = tf.assign(hidden1_weights, original_weights)
assign_hidden1_biases = tf.assign(hidden1_biases, original_biases)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
sess.run(assign_hidden1_weights, feed_dict={original_weights: original_w})
sess.run(assign_hidden1_biases, feed_dict={original_biases: original_b})
# [...] Train the model on your new task
print(hidden1.eval(feed_dict={X: [[10.0, 11.0]]}))
[[ 61. 83. 105.]]
Note that we could also get a handle on the variables using get_collection() and specifying the scope:
In [65]:
tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope="hidden1")
Out[65]:
[<tf.Variable 'hidden1/kernel:0' shape=(2, 3) dtype=float32_ref>, <tf.Variable 'hidden1/bias:0' shape=(3,) dtype=float32_ref>]
Or we could use the graph's get_tensor_by_name() method:
In [66]:
tf.get_default_graph().get_tensor_by_name("hidden1/kernel:0")
Out[66]:
<tf.Tensor 'hidden1/kernel:0' shape=(2, 3) dtype=float32_ref>
In [67]:
tf.get_default_graph().get_tensor_by_name("hidden1/bias:0")
Out[67]:
<tf.Tensor 'hidden1/bias:0' shape=(3,) dtype=float32_ref>
Freezing the Lower Layers¶
In [68]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300 # reused
n_hidden2 = 50 # reused
n_hidden3 = 50 # reused
n_hidden4 = 20 # new!
n_outputs = 10 # new!
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1") # reused
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2") # reused
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, name="hidden3") # reused
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name="hidden4") # new!
logits = tf.layers.dense(hidden4, n_outputs, name="outputs") # new!
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
In [69]:
with tf.name_scope("train"): # not shown in the book
optimizer = tf.train.GradientDescentOptimizer(learning_rate) # not shown
train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
scope="hidden[34]|outputs")
training_op = optimizer.minimize(loss, var_list=train_vars)
In [70]:
init = tf.global_variables_initializer()
new_saver = tf.train.Saver()
In [71]:
reuse_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="hidden[123]") # regular expression
restore_saver = tf.train.Saver(reuse_vars) # to restore layers 1-3
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.903 1 Validation accuracy: 0.9302 2 Validation accuracy: 0.939 3 Validation accuracy: 0.9406 4 Validation accuracy: 0.9458 5 Validation accuracy: 0.9486 6 Validation accuracy: 0.95 7 Validation accuracy: 0.9528 8 Validation accuracy: 0.9532 9 Validation accuracy: 0.953 10 Validation accuracy: 0.9552 11 Validation accuracy: 0.9558 12 Validation accuracy: 0.9546 13 Validation accuracy: 0.9548 14 Validation accuracy: 0.9558 15 Validation accuracy: 0.9578 16 Validation accuracy: 0.9576 17 Validation accuracy: 0.9586 18 Validation accuracy: 0.9586 19 Validation accuracy: 0.958
In [72]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300 # reused
n_hidden2 = 50 # reused
n_hidden3 = 50 # reused
n_hidden4 = 20 # new!
n_outputs = 10 # new!
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
In [73]:
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu,
name="hidden1") # reused frozen
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu,
name="hidden2") # reused frozen
hidden2_stop = tf.stop_gradient(hidden2)
hidden3 = tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu,
name="hidden3") # reused, not frozen
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu,
name="hidden4") # new!
logits = tf.layers.dense(hidden4, n_outputs, name="outputs") # new!
In [74]:
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
The training code is exactly the same as earlier:
In [75]:
reuse_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="hidden[123]") # regular expression
restore_saver = tf.train.Saver(reuse_vars) # to restore layers 1-3
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_model_final.ckpt")
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.9184 1 Validation accuracy: 0.9406 2 Validation accuracy: 0.9468 3 Validation accuracy: 0.9508 4 Validation accuracy: 0.951 5 Validation accuracy: 0.953 6 Validation accuracy: 0.9534 7 Validation accuracy: 0.9554 8 Validation accuracy: 0.9562 9 Validation accuracy: 0.9574 10 Validation accuracy: 0.957 11 Validation accuracy: 0.9582 12 Validation accuracy: 0.9574 13 Validation accuracy: 0.96 14 Validation accuracy: 0.9596 15 Validation accuracy: 0.9582 16 Validation accuracy: 0.9592 17 Validation accuracy: 0.9602 18 Validation accuracy: 0.9606 19 Validation accuracy: 0.9598
Caching the Frozen Layers¶
In [76]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300 # reused
n_hidden2 = 50 # reused
n_hidden3 = 50 # reused
n_hidden4 = 20 # new!
n_outputs = 10 # new!
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu,
name="hidden1") # reused frozen
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu,
name="hidden2") # reused frozen & cached
hidden2_stop = tf.stop_gradient(hidden2)
hidden3 = tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu,
name="hidden3") # reused, not frozen
hidden4 = tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu,
name="hidden4") # new!
logits = tf.layers.dense(hidden4, n_outputs, name="outputs") # new!
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
In [77]:
reuse_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="hidden[123]") # regular expression
restore_saver = tf.train.Saver(reuse_vars) # to restore layers 1-3
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [78]:
import numpy as np
n_batches = len(X_train) // batch_size
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_model_final.ckpt")
h2_cache = sess.run(hidden2, feed_dict={X: X_train})
h2_cache_valid = sess.run(hidden2, feed_dict={X: X_valid}) # not shown in the book
for epoch in range(n_epochs):
shuffled_idx = np.random.permutation(len(X_train))
hidden2_batches = np.array_split(h2_cache[shuffled_idx], n_batches)
y_batches = np.array_split(y_train[shuffled_idx], n_batches)
for hidden2_batch, y_batch in zip(hidden2_batches, y_batches):
sess.run(training_op, feed_dict={hidden2:hidden2_batch, y:y_batch})
accuracy_val = accuracy.eval(feed_dict={hidden2: h2_cache_valid, # not shown
y: y_valid}) # not shown
print(epoch, "Validation accuracy:", accuracy_val) # not shown
save_path = saver.save(sess, "./my_new_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_model_final.ckpt 0 Validation accuracy: 0.9184 1 Validation accuracy: 0.9406 2 Validation accuracy: 0.9468 3 Validation accuracy: 0.9508 4 Validation accuracy: 0.951 5 Validation accuracy: 0.953 6 Validation accuracy: 0.9534 7 Validation accuracy: 0.9554 8 Validation accuracy: 0.9562 9 Validation accuracy: 0.9574 10 Validation accuracy: 0.957 11 Validation accuracy: 0.9582 12 Validation accuracy: 0.9574 13 Validation accuracy: 0.96 14 Validation accuracy: 0.9596 15 Validation accuracy: 0.9582 16 Validation accuracy: 0.9592 17 Validation accuracy: 0.9602 18 Validation accuracy: 0.9606 19 Validation accuracy: 0.9598
Faster Optimizers¶
Momentum optimization¶
In [79]:
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
momentum=0.9)
Nesterov Accelerated Gradient¶
In [80]:
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
momentum=0.9, use_nesterov=True)
AdaGrad¶
In [81]:
optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)
RMSProp¶
In [82]:
optimizer = tf.train.RMSPropOptimizer(learning_rate=learning_rate,
momentum=0.9, decay=0.9, epsilon=1e-10)
Adam Optimization¶
In [83]:
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
Learning Rate Scheduling¶
In [84]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 50
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2")
logits = tf.layers.dense(hidden2, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
In [85]:
with tf.name_scope("train"): # not shown in the book
initial_learning_rate = 0.1
decay_steps = 10000
decay_rate = 1/10
global_step = tf.Variable(0, trainable=False, name="global_step")
learning_rate = tf.train.exponential_decay(initial_learning_rate, global_step,
decay_steps, decay_rate)
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)
training_op = optimizer.minimize(loss, global_step=global_step)
In [86]:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [87]:
n_epochs = 5
batch_size = 50
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.963 1 Validation accuracy: 0.9712 2 Validation accuracy: 0.9758 3 Validation accuracy: 0.9802 4 Validation accuracy: 0.9814
Avoiding Overfitting Through Regularization¶
$\ell_1$ and $\ell_2$ regularization¶
Let's implement $\ell_1$ regularization manually. First, we create the model, as usual (with just one hidden layer this time, for simplicity):
In [88]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
logits = tf.layers.dense(hidden1, n_outputs, name="outputs")
Next, we get a handle on the layer weights, and we compute the total loss, which is equal to the sum of the usual cross entropy loss and the $\ell_1$ loss (i.e., the absolute values of the weights):
In [89]:
W1 = tf.get_default_graph().get_tensor_by_name("hidden1/kernel:0")
W2 = tf.get_default_graph().get_tensor_by_name("outputs/kernel:0")
scale = 0.001 # l1 regularization hyperparameter
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
logits=logits)
base_loss = tf.reduce_mean(xentropy, name="avg_xentropy")
reg_losses = tf.reduce_sum(tf.abs(W1)) + tf.reduce_sum(tf.abs(W2))
loss = tf.add(base_loss, scale * reg_losses, name="loss")
The rest is just as usual:
In [90]:
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
learning_rate = 0.01
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [91]:
n_epochs = 20
batch_size = 200
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.8282 1 Validation accuracy: 0.8642 2 Validation accuracy: 0.8822 3 Validation accuracy: 0.8898 4 Validation accuracy: 0.8946 5 Validation accuracy: 0.898 6 Validation accuracy: 0.9022 7 Validation accuracy: 0.9044 8 Validation accuracy: 0.9054 9 Validation accuracy: 0.9072 10 Validation accuracy: 0.9076 11 Validation accuracy: 0.9098 12 Validation accuracy: 0.9088 13 Validation accuracy: 0.9088 14 Validation accuracy: 0.9104 15 Validation accuracy: 0.909 16 Validation accuracy: 0.9086 17 Validation accuracy: 0.908 18 Validation accuracy: 0.9082 19 Validation accuracy: 0.9062
Alternatively, we can pass a regularization function to the tf.layers.dense() function, which will use it to create operations that will compute the regularization loss, and it adds these operations to the collection of regularization losses. The beginning is the same as above:
In [92]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_hidden1 = 300
n_hidden2 = 50
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
Next, we will use Python's partial() function to avoid repeating the same arguments over and over again. Note that we set the kernel_regularizer argument:
In [93]:
scale = 0.001
In [94]:
my_dense_layer = partial(
tf.layers.dense, activation=tf.nn.relu,
kernel_regularizer=tf.contrib.layers.l1_regularizer(scale))
with tf.name_scope("dnn"):
hidden1 = my_dense_layer(X, n_hidden1, name="hidden1")
hidden2 = my_dense_layer(hidden1, n_hidden2, name="hidden2")
logits = my_dense_layer(hidden2, n_outputs, activation=None,
name="outputs")
WARNING:tensorflow:From /home/ageron/.virtualenvs/ml/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives.
Next we must add the regularization losses to the base loss:
In [95]:
with tf.name_scope("loss"): # not shown in the book
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits( # not shown
labels=y, logits=logits) # not shown
base_loss = tf.reduce_mean(xentropy, name="avg_xentropy") # not shown
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([base_loss] + reg_losses, name="loss")
And the rest is the same as usual:
In [96]:
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
learning_rate = 0.01
with tf.name_scope("train"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [97]:
n_epochs = 20
batch_size = 200
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.8126 1 Validation accuracy: 0.8702 2 Validation accuracy: 0.8884 3 Validation accuracy: 0.8982 4 Validation accuracy: 0.902 5 Validation accuracy: 0.9074 6 Validation accuracy: 0.9088 7 Validation accuracy: 0.9118 8 Validation accuracy: 0.9126 9 Validation accuracy: 0.9148 10 Validation accuracy: 0.916 11 Validation accuracy: 0.9162 12 Validation accuracy: 0.9158 13 Validation accuracy: 0.917 14 Validation accuracy: 0.9174 15 Validation accuracy: 0.9174 16 Validation accuracy: 0.9172 17 Validation accuracy: 0.9186 18 Validation accuracy: 0.9186 19 Validation accuracy: 0.9182
Dropout¶
Note: the book uses tf.contrib.layers.dropout() rather than tf.layers.dropout() (which did not exist when this chapter was written). It is now preferable to use tf.layers.dropout(), because anything in the contrib module may change or be deleted without notice. The tf.layers.dropout() function is almost identical to the tf.contrib.layers.dropout() function, except for a few minor differences. Most importantly:
- you must specify the dropout rate (
rate) rather than the keep probability (keep_prob), whererateis simply equal to1 - keep_prob, - the
is_trainingparameter is renamed totraining.
In [98]:
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
In [99]:
training = tf.placeholder_with_default(False, shape=(), name='training')
dropout_rate = 0.5 # == 1 - keep_prob
X_drop = tf.layers.dropout(X, dropout_rate, training=training)
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X_drop, n_hidden1, activation=tf.nn.relu,
name="hidden1")
hidden1_drop = tf.layers.dropout(hidden1, dropout_rate, training=training)
hidden2 = tf.layers.dense(hidden1_drop, n_hidden2, activation=tf.nn.relu,
name="hidden2")
hidden2_drop = tf.layers.dropout(hidden2, dropout_rate, training=training)
logits = tf.layers.dense(hidden2_drop, n_outputs, name="outputs")
In [100]:
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("train"):
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
saver = tf.train.Saver()
In [101]:
n_epochs = 20
batch_size = 50
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch, training: True})
accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})
print(epoch, "Validation accuracy:", accuracy_val)
save_path = saver.save(sess, "./my_model_final.ckpt")
0 Validation accuracy: 0.9602 1 Validation accuracy: 0.9712 2 Validation accuracy: 0.9712 3 Validation accuracy: 0.975 4 Validation accuracy: 0.976 5 Validation accuracy: 0.9782 6 Validation accuracy: 0.981 7 Validation accuracy: 0.9808 8 Validation accuracy: 0.982 9 Validation accuracy: 0.9816 10 Validation accuracy: 0.9824 11 Validation accuracy: 0.9836 12 Validation accuracy: 0.9798 13 Validation accuracy: 0.9832 14 Validation accuracy: 0.982 15 Validation accuracy: 0.9834 16 Validation accuracy: 0.9832 17 Validation accuracy: 0.9842 18 Validation accuracy: 0.9832 19 Validation accuracy: 0.984
Max norm¶
Let's go back to a plain and simple neural net for MNIST with just 2 hidden layers:
In [102]:
reset_graph()
n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 50
n_outputs = 10
learning_rate = 0.01
momentum = 0.9
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name="hidden2")
logits = tf.layers.dense(hidden2, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("train"):
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
Next, let's get a handle on the first hidden layer's weight and create an operation that will compute the clipped weights using the clip_by_norm() function. Then we create an assignment operation to assign the clipped weights to the weights variable:
In [103]:
threshold = 1.0
weights = tf.get_default_graph().get_tensor_by_name("hidden1/kernel:0")
clipped_weights = tf.clip_by_norm(weights, clip_norm=threshold, axes=1)
clip_weights = tf.assign(weights, clipped_weights)
We can do this as well for the second hidden layer:
In [104]:
weights2 = tf.get_default_graph().get_tensor_by_name("hidden2/kernel:0")
clipped_weights2 = tf.clip_by_norm(weights2, clip_norm=threshold, axes=1)
clip_weights2 = tf.assign(weights2, clipped_weights2)
Let's add an initializer and a saver:
In [105]:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
And now we can train the model. It's pretty much as usual, except that right after running the training_op, we run the clip_weights and clip_weights2 operations:
In [106]:
n_epochs = 20
batch_size = 50
In [107]:
with tf.Session() as sess: # not shown in the book
init.run() # not shown
for epoch in range(n_epochs): # not shown
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size): # not shown
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
clip_weights.eval()
clip_weights2.eval() # not shown
acc_valid = accuracy.eval(feed_dict={X: X_valid, y: y_valid}) # not shown
print(epoch, "Validation accuracy:", acc_valid) # not shown
save_path = saver.save(sess, "./my_model_final.ckpt") # not shown
0 Validation accuracy: 0.9578 1 Validation accuracy: 0.9696 2 Validation accuracy: 0.97 3 Validation accuracy: 0.9786 4 Validation accuracy: 0.978 5 Validation accuracy: 0.9792 6 Validation accuracy: 0.9822 7 Validation accuracy: 0.9814 8 Validation accuracy: 0.9796 9 Validation accuracy: 0.9792 10 Validation accuracy: 0.9814 11 Validation accuracy: 0.9826 12 Validation accuracy: 0.9812 13 Validation accuracy: 0.9838 14 Validation accuracy: 0.9818 15 Validation accuracy: 0.9824 16 Validation accuracy: 0.9824 17 Validation accuracy: 0.982 18 Validation accuracy: 0.983 19 Validation accuracy: 0.9828
The implementation above is straightforward and it works fine, but it is a bit messy. A better approach is to define a max_norm_regularizer() function:
In [108]:
def max_norm_regularizer(threshold, axes=1, name="max_norm",
collection="max_norm"):
def max_norm(weights):
clipped = tf.clip_by_norm(weights, clip_norm=threshold, axes=axes)
clip_weights = tf.assign(weights, clipped, name=name)
tf.add_to_collection(collection, clip_weights)
return None # there is no regularization loss term
return max_norm
Then you can call this function to get a max norm regularizer (with the threshold you want). When you create a hidden layer, you can pass this regularizer to the kernel_regularizer argument:
In [109]:
reset_graph()
n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 50
n_outputs = 10
learning_rate = 0.01
momentum = 0.9
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
In [110]:
max_norm_reg = max_norm_regularizer(threshold=1.0)
with tf.name_scope("dnn"):
hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu,
kernel_regularizer=max_norm_reg, name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu,
kernel_regularizer=max_norm_reg, name="hidden2")
logits = tf.layers.dense(hidden2, n_outputs, name="outputs")
In [111]:
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("train"):
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
saver = tf.train.Saver()
Training is as usual, except you must run the weights clipping operations after each training operation:
In [112]:
n_epochs = 20
batch_size = 50
In [113]:
clip_all_weights = tf.get_collection("max_norm")
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for X_batch, y_batch in shuffle_batch(X_train, y_train, batch_size):
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
sess.run(clip_all_weights)
acc_valid = accuracy.eval(feed_dict={X: X_valid, y: y_valid}) # not shown
print(epoch, "Validation accuracy:", acc_valid) # not shown
save_path = saver.save(sess, "./my_model_final.ckpt") # not shown
0 Validation accuracy: 0.9588 1 Validation accuracy: 0.9718 2 Validation accuracy: 0.9708 3 Validation accuracy: 0.977 4 Validation accuracy: 0.9744 5 Validation accuracy: 0.98 6 Validation accuracy: 0.981 7 Validation accuracy: 0.9814 8 Validation accuracy: 0.981 9 Validation accuracy: 0.9808 10 Validation accuracy: 0.9828 11 Validation accuracy: 0.9838 12 Validation accuracy: 0.982 13 Validation accuracy: 0.9836 14 Validation accuracy: 0.9828 15 Validation accuracy: 0.9842 16 Validation accuracy: 0.9838 17 Validation accuracy: 0.9832 18 Validation accuracy: 0.9844 19 Validation accuracy: 0.9842
Exercise solutions¶
1. to 7.¶
See appendix A.
8. Deep Learning¶
8.1.¶
Exercise: Build a DNN with five hidden layers of 100 neurons each, He initialization, and the ELU activation function.
We will need similar DNNs in the next exercises, so let's create a function to build this DNN:
In [114]:
he_init = tf.variance_scaling_initializer()
def dnn(inputs, n_hidden_layers=5, n_neurons=100, name=None,
activation=tf.nn.elu, initializer=he_init):
with tf.variable_scope(name, "dnn"):
for layer in range(n_hidden_layers):
inputs = tf.layers.dense(inputs, n_neurons, activation=activation,
kernel_initializer=initializer,
name="hidden%d" % (layer + 1))
return inputs
In [115]:
n_inputs = 28 * 28 # MNIST
n_outputs = 5
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
dnn_outputs = dnn(X)
logits = tf.layers.dense(dnn_outputs, n_outputs, kernel_initializer=he_init, name="logits")
Y_proba = tf.nn.softmax(logits, name="Y_proba")
8.2.¶
Exercise: Using Adam optimization and early stopping, try training it on MNIST but only on digits 0 to 4, as we will use transfer learning for digits 5 to 9 in the next exercise. You will need a softmax output layer with five neurons, and as always make sure to save checkpoints at regular intervals and save the final model so you can reuse it later.
Let's complete the graph with the cost function, the training op, and all the other usual components:
In [116]:
learning_rate = 0.01
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss, name="training_op")
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
init = tf.global_variables_initializer()
saver = tf.train.Saver()
Now let's create the training set, validation and test set (we need the validation set to implement early stopping):
In [117]:
X_train1 = X_train[y_train < 5]
y_train1 = y_train[y_train < 5]
X_valid1 = X_valid[y_valid < 5]
y_valid1 = y_valid[y_valid < 5]
X_test1 = X_test[y_test < 5]
y_test1 = y_test[y_test < 5]
In [118]:
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train1))
for rnd_indices in np.array_split(rnd_idx, len(X_train1) // batch_size):
X_batch, y_batch = X_train1[rnd_indices], y_train1[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid1, y: y_valid1})
if loss_val < best_loss:
save_path = saver.save(sess, "./my_mnist_model_0_to_4.ckpt")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
with tf.Session() as sess:
saver.restore(sess, "./my_mnist_model_0_to_4.ckpt")
acc_test = accuracy.eval(feed_dict={X: X_test1, y: y_test1})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
0 Validation loss: 0.466011 Best loss: 0.466011 Accuracy: 77.76% 1 Validation loss: 0.100298 Best loss: 0.100298 Accuracy: 97.77% 2 Validation loss: 1.603312 Best loss: 0.100298 Accuracy: 18.73% 3 Validation loss: 1.759563 Best loss: 0.100298 Accuracy: 19.08% 4 Validation loss: 1.640234 Best loss: 0.100298 Accuracy: 22.01% 5 Validation loss: 1.640949 Best loss: 0.100298 Accuracy: 22.01% 6 Validation loss: 1.670187 Best loss: 0.100298 Accuracy: 18.73% 7 Validation loss: 1.764533 Best loss: 0.100298 Accuracy: 22.01% 8 Validation loss: 1.700626 Best loss: 0.100298 Accuracy: 19.27% 9 Validation loss: 1.767139 Best loss: 0.100298 Accuracy: 20.91% 10 Validation loss: 1.629366 Best loss: 0.100298 Accuracy: 22.01% 11 Validation loss: 1.812608 Best loss: 0.100298 Accuracy: 22.01% 12 Validation loss: 1.675928 Best loss: 0.100298 Accuracy: 18.73% 13 Validation loss: 1.633258 Best loss: 0.100298 Accuracy: 20.91% 14 Validation loss: 1.652906 Best loss: 0.100298 Accuracy: 20.91% 15 Validation loss: 1.635938 Best loss: 0.100298 Accuracy: 20.91% 16 Validation loss: 1.718920 Best loss: 0.100298 Accuracy: 19.08% 17 Validation loss: 1.682459 Best loss: 0.100298 Accuracy: 19.27% 18 Validation loss: 1.675367 Best loss: 0.100298 Accuracy: 18.73% 19 Validation loss: 1.645803 Best loss: 0.100298 Accuracy: 19.08% 20 Validation loss: 1.722334 Best loss: 0.100298 Accuracy: 22.01% 21 Validation loss: 1.656415 Best loss: 0.100298 Accuracy: 22.01% Early stopping! INFO:tensorflow:Restoring parameters from ./my_mnist_model_0_to_4.ckpt Final test accuracy: 97.92%
We get 98.05% accuracy on the test set. That's not too bad, but let's see if we can do better by tuning the hyperparameters.
8.3.¶
Exercise: Tune the hyperparameters using cross-validation and see what precision you can achieve.
Let's create a DNNClassifier class, compatible with Scikit-Learn's RandomizedSearchCV class, to perform hyperparameter tuning. Here are the key points of this implementation:
- the
__init__()method (constructor) does nothing more than create instance variables for each of the hyperparameters. - the
fit()method creates the graph, starts a session and trains the model:- it calls the
_build_graph()method to build the graph (much lile the graph we defined earlier). Once this method is done creating the graph, it saves all the important operations as instance variables for easy access by other methods. - the
_dnn()method builds the hidden layers, just like thednn()function above, but also with support for batch normalization and dropout (for the next exercises). - if the
fit()method is given a validation set (X_validandy_valid), then it implements early stopping. This implementation does not save the best model to disk, but rather to memory: it uses the_get_model_params()method to get all the graph's variables and their values, and the_restore_model_params()method to restore the variable values (of the best model found). This trick helps speed up training. - After the
fit()method has finished training the model, it keeps the session open so that predictions can be made quickly, without having to save a model to disk and restore it for every prediction. You can close the session by calling theclose_session()method.
- it calls the
- the
predict_proba()method uses the trained model to predict the class probabilities. - the
predict()method callspredict_proba()and returns the class with the highest probability, for each instance.
In [119]:
from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.exceptions import NotFittedError
class DNNClassifier(BaseEstimator, ClassifierMixin):
def __init__(self, n_hidden_layers=5, n_neurons=100, optimizer_class=tf.train.AdamOptimizer,
learning_rate=0.01, batch_size=20, activation=tf.nn.elu, initializer=he_init,
batch_norm_momentum=None, dropout_rate=None, random_state=None):
"""Initialize the DNNClassifier by simply storing all the hyperparameters."""
self.n_hidden_layers = n_hidden_layers
self.n_neurons = n_neurons
self.optimizer_class = optimizer_class
self.learning_rate = learning_rate
self.batch_size = batch_size
self.activation = activation
self.initializer = initializer
self.batch_norm_momentum = batch_norm_momentum
self.dropout_rate = dropout_rate
self.random_state = random_state
self._session = None
def _dnn(self, inputs):
"""Build the hidden layers, with support for batch normalization and dropout."""
for layer in range(self.n_hidden_layers):
if self.dropout_rate:
inputs = tf.layers.dropout(inputs, self.dropout_rate, training=self._training)
inputs = tf.layers.dense(inputs, self.n_neurons,
kernel_initializer=self.initializer,
name="hidden%d" % (layer + 1))
if self.batch_norm_momentum:
inputs = tf.layers.batch_normalization(inputs, momentum=self.batch_norm_momentum,
training=self._training)
inputs = self.activation(inputs, name="hidden%d_out" % (layer + 1))
return inputs
def _build_graph(self, n_inputs, n_outputs):
"""Build the same model as earlier"""
if self.random_state is not None:
tf.set_random_seed(self.random_state)
np.random.seed(self.random_state)
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
if self.batch_norm_momentum or self.dropout_rate:
self._training = tf.placeholder_with_default(False, shape=(), name='training')
else:
self._training = None
dnn_outputs = self._dnn(X)
logits = tf.layers.dense(dnn_outputs, n_outputs, kernel_initializer=he_init, name="logits")
Y_proba = tf.nn.softmax(logits, name="Y_proba")
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
optimizer = self.optimizer_class(learning_rate=self.learning_rate)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
init = tf.global_variables_initializer()
saver = tf.train.Saver()
# Make the important operations available easily through instance variables
self._X, self._y = X, y
self._Y_proba, self._loss = Y_proba, loss
self._training_op, self._accuracy = training_op, accuracy
self._init, self._saver = init, saver
def close_session(self):
if self._session:
self._session.close()
def _get_model_params(self):
"""Get all variable values (used for early stopping, faster than saving to disk)"""
with self._graph.as_default():
gvars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
return {gvar.op.name: value for gvar, value in zip(gvars, self._session.run(gvars))}
def _restore_model_params(self, model_params):
"""Set all variables to the given values (for early stopping, faster than loading from disk)"""
gvar_names = list(model_params.keys())
assign_ops = {gvar_name: self._graph.get_operation_by_name(gvar_name + "/Assign")
for gvar_name in gvar_names}
init_values = {gvar_name: assign_op.inputs[1] for gvar_name, assign_op in assign_ops.items()}
feed_dict = {init_values[gvar_name]: model_params[gvar_name] for gvar_name in gvar_names}
self._session.run(assign_ops, feed_dict=feed_dict)
def fit(self, X, y, n_epochs=100, X_valid=None, y_valid=None):
"""Fit the model to the training set. If X_valid and y_valid are provided, use early stopping."""
self.close_session()
# infer n_inputs and n_outputs from the training set.
n_inputs = X.shape[1]
self.classes_ = np.unique(y)
n_outputs = len(self.classes_)
# Translate the labels vector to a vector of sorted class indices, containing
# integers from 0 to n_outputs - 1.
# For example, if y is equal to [8, 8, 9, 5, 7, 6, 6, 6], then the sorted class
# labels (self.classes_) will be equal to [5, 6, 7, 8, 9], and the labels vector
# will be translated to [3, 3, 4, 0, 2, 1, 1, 1]
self.class_to_index_ = {label: index
for index, label in enumerate(self.classes_)}
y = np.array([self.class_to_index_[label]
for label in y], dtype=np.int32)
self._graph = tf.Graph()
with self._graph.as_default():
self._build_graph(n_inputs, n_outputs)
# extra ops for batch normalization
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
# needed in case of early stopping
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
best_params = None
# Now train the model!
self._session = tf.Session(graph=self._graph)
with self._session.as_default() as sess:
self._init.run()
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X))
for rnd_indices in np.array_split(rnd_idx, len(X) // self.batch_size):
X_batch, y_batch = X[rnd_indices], y[rnd_indices]
feed_dict = {self._X: X_batch, self._y: y_batch}
if self._training is not None:
feed_dict[self._training] = True
sess.run(self._training_op, feed_dict=feed_dict)
if extra_update_ops:
sess.run(extra_update_ops, feed_dict=feed_dict)
if X_valid is not None and y_valid is not None:
loss_val, acc_val = sess.run([self._loss, self._accuracy],
feed_dict={self._X: X_valid,
self._y: y_valid})
if loss_val < best_loss:
best_params = self._get_model_params()
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
else:
loss_train, acc_train = sess.run([self._loss, self._accuracy],
feed_dict={self._X: X_batch,
self._y: y_batch})
print("{}\tLast training batch loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_train, acc_train * 100))
# If we used early stopping then rollback to the best model found
if best_params:
self._restore_model_params(best_params)
return self
def predict_proba(self, X):
if not self._session:
raise NotFittedError("This %s instance is not fitted yet" % self.__class__.__name__)
with self._session.as_default() as sess:
return self._Y_proba.eval(feed_dict={self._X: X})
def predict(self, X):
class_indices = np.argmax(self.predict_proba(X), axis=1)
return np.array([[self.classes_[class_index]]
for class_index in class_indices], np.int32)
def save(self, path):
self._saver.save(self._session, path)
Let's see if we get the exact same accuracy as earlier using this class (without dropout or batch norm):
In [120]:
dnn_clf = DNNClassifier(random_state=42)
dnn_clf.fit(X_train1, y_train1, n_epochs=1000, X_valid=X_valid1, y_valid=y_valid1)
0 Validation loss: 0.183419 Best loss: 0.183419 Accuracy: 96.64% 1 Validation loss: 1.649454 Best loss: 0.183419 Accuracy: 19.27% 2 Validation loss: 1.670295 Best loss: 0.183419 Accuracy: 20.91% 3 Validation loss: 1.785215 Best loss: 0.183419 Accuracy: 22.01% 4 Validation loss: 1.667377 Best loss: 0.183419 Accuracy: 22.01% 5 Validation loss: 1.654941 Best loss: 0.183419 Accuracy: 22.01% 6 Validation loss: 1.681062 Best loss: 0.183419 Accuracy: 18.73% 7 Validation loss: 1.779103 Best loss: 0.183419 Accuracy: 22.01% 8 Validation loss: 1.699479 Best loss: 0.183419 Accuracy: 19.27% 9 Validation loss: 1.767774 Best loss: 0.183419 Accuracy: 20.91% 10 Validation loss: 1.629342 Best loss: 0.183419 Accuracy: 22.01% 11 Validation loss: 1.812649 Best loss: 0.183419 Accuracy: 22.01% 12 Validation loss: 1.675937 Best loss: 0.183419 Accuracy: 18.73% 13 Validation loss: 1.633258 Best loss: 0.183419 Accuracy: 20.91% 14 Validation loss: 1.652906 Best loss: 0.183419 Accuracy: 20.91% 15 Validation loss: 1.635939 Best loss: 0.183419 Accuracy: 20.91% 16 Validation loss: 1.718920 Best loss: 0.183419 Accuracy: 19.08% 17 Validation loss: 1.682459 Best loss: 0.183419 Accuracy: 19.27% 18 Validation loss: 1.675367 Best loss: 0.183419 Accuracy: 18.73% 19 Validation loss: 1.645803 Best loss: 0.183419 Accuracy: 19.08% 20 Validation loss: 1.722334 Best loss: 0.183419 Accuracy: 22.01% 21 Validation loss: 1.656415 Best loss: 0.183419 Accuracy: 22.01% Early stopping!
Out[120]:
DNNClassifier(activation=<function elu at 0x7f95d738dbf8>,
batch_norm_momentum=None, batch_size=20, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=100,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42)
The model is trained, let's see if it gets the same accuracy as earlier:
In [121]:
from sklearn.metrics import accuracy_score
y_pred = dnn_clf.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[121]:
0.9710060323020043
Yep! Working fine. Now we can use Scikit-Learn's RandomizedSearchCV class to search for better hyperparameters (this may take over an hour, depending on your system):
In [122]:
from sklearn.model_selection import RandomizedSearchCV
def leaky_relu(alpha=0.01):
def parametrized_leaky_relu(z, name=None):
return tf.maximum(alpha * z, z, name=name)
return parametrized_leaky_relu
param_distribs = {
"n_neurons": [10, 30, 50, 70, 90, 100, 120, 140, 160],
"batch_size": [10, 50, 100, 500],
"learning_rate": [0.01, 0.02, 0.05, 0.1],
"activation": [tf.nn.relu, tf.nn.elu, leaky_relu(alpha=0.01), leaky_relu(alpha=0.1)],
# you could also try exploring different numbers of hidden layers, different optimizers, etc.
#"n_hidden_layers": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
#"optimizer_class": [tf.train.AdamOptimizer, partial(tf.train.MomentumOptimizer, momentum=0.95)],
}
rnd_search = RandomizedSearchCV(DNNClassifier(random_state=42), param_distribs, n_iter=50,
random_state=42, verbose=2)
rnd_search.fit(X_train1, y_train1, X_valid=X_valid1, y_valid=y_valid1, n_epochs=1000)
# If you have Scikit-Learn 0.18 or earlier, you should upgrade, or use the fit_params argument:
# fit_params={"X_valid": X_valid1, "y_valid": y_valid1, "n_epochs": 1000}
# rnd_search = RandomizedSearchCV(DNNClassifier(random_state=42), param_distribs, n_iter=50,
# fit_params=fit_params, random_state=42, verbose=2)
# rnd_search.fit(X_train1, y_train1)
Fitting 3 folds for each of 50 candidates, totalling 150 fits [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05 0 Validation loss: 0.228535 Best loss: 0.228535 Accuracy: 94.21% 1 Validation loss: 0.178857 Best loss: 0.178857 Accuracy: 94.21% 2 Validation loss: 0.169731 Best loss: 0.169731 Accuracy: 95.50% 3 Validation loss: 0.124046 Best loss: 0.124046 Accuracy: 96.36% 4 Validation loss: 0.122978 Best loss: 0.122978 Accuracy: 96.52% 5 Validation loss: 0.165714 Best loss: 0.122978 Accuracy: 96.33% 6 Validation loss: 0.162516 Best loss: 0.122978 Accuracy: 96.21% 7 Validation loss: 0.949882 Best loss: 0.122978 Accuracy: 60.79% 8 Validation loss: 1.071238 Best loss: 0.122978 Accuracy: 50.86% 9 Validation loss: 0.982995 Best loss: 0.122978 Accuracy: 58.60% 10 Validation loss: 1.095278 Best loss: 0.122978 Accuracy: 48.12% 11 Validation loss: 0.925425 Best loss: 0.122978 Accuracy: 52.66% 12 Validation loss: 1.038524 Best loss: 0.122978 Accuracy: 51.99% 13 Validation loss: 1.381219 Best loss: 0.122978 Accuracy: 32.53% 14 Validation loss: 1.199629 Best loss: 0.122978 Accuracy: 43.67% 15 Validation loss: 1.019348 Best loss: 0.122978 Accuracy: 54.53% 16 Validation loss: 0.891379 Best loss: 0.122978 Accuracy: 59.30% 17 Validation loss: 0.927038 Best loss: 0.122978 Accuracy: 56.06% 18 Validation loss: 0.917532 Best loss: 0.122978 Accuracy: 60.79% 19 Validation loss: 0.866032 Best loss: 0.122978 Accuracy: 53.67% 20 Validation loss: 0.962458 Best loss: 0.122978 Accuracy: 55.63% 21 Validation loss: 0.898072 Best loss: 0.122978 Accuracy: 55.67% 22 Validation loss: 1.037736 Best loss: 0.122978 Accuracy: 59.58% 23 Validation loss: 0.917886 Best loss: 0.122978 Accuracy: 60.67% 24 Validation loss: 1.042268 Best loss: 0.122978 Accuracy: 55.16% 25 Validation loss: 1.215994 Best loss: 0.122978 Accuracy: 40.58% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 10.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05
[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 10.8s remaining: 0.0s
0 Validation loss: 0.157979 Best loss: 0.157979 Accuracy: 95.35% 1 Validation loss: 0.166130 Best loss: 0.157979 Accuracy: 96.05% 2 Validation loss: 0.136959 Best loss: 0.136959 Accuracy: 96.09% 3 Validation loss: 0.668840 Best loss: 0.136959 Accuracy: 86.86% 4 Validation loss: 1.629821 Best loss: 0.136959 Accuracy: 22.01% 5 Validation loss: 1.615475 Best loss: 0.136959 Accuracy: 22.01% 6 Validation loss: 1.630031 Best loss: 0.136959 Accuracy: 18.73% 7 Validation loss: 1.648450 Best loss: 0.136959 Accuracy: 18.73% 8 Validation loss: 1.610620 Best loss: 0.136959 Accuracy: 22.01% 9 Validation loss: 1.616665 Best loss: 0.136959 Accuracy: 22.01% 10 Validation loss: 1.615181 Best loss: 0.136959 Accuracy: 22.01% 11 Validation loss: 1.610208 Best loss: 0.136959 Accuracy: 22.01% 12 Validation loss: 1.622679 Best loss: 0.136959 Accuracy: 19.27% 13 Validation loss: 1.613467 Best loss: 0.136959 Accuracy: 19.27% 14 Validation loss: 1.610566 Best loss: 0.136959 Accuracy: 22.01% 15 Validation loss: 1.624033 Best loss: 0.136959 Accuracy: 22.01% 16 Validation loss: 1.622954 Best loss: 0.136959 Accuracy: 19.27% 17 Validation loss: 1.614533 Best loss: 0.136959 Accuracy: 19.08% 18 Validation loss: 1.611469 Best loss: 0.136959 Accuracy: 22.01% 19 Validation loss: 1.612612 Best loss: 0.136959 Accuracy: 22.01% 20 Validation loss: 1.608505 Best loss: 0.136959 Accuracy: 22.01% 21 Validation loss: 1.631395 Best loss: 0.136959 Accuracy: 22.01% 22 Validation loss: 1.621655 Best loss: 0.136959 Accuracy: 19.27% 23 Validation loss: 1.628181 Best loss: 0.136959 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 9.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05 0 Validation loss: 0.144850 Best loss: 0.144850 Accuracy: 96.40% 1 Validation loss: 0.177231 Best loss: 0.144850 Accuracy: 96.44% 2 Validation loss: 0.313004 Best loss: 0.144850 Accuracy: 92.38% 3 Validation loss: 0.262883 Best loss: 0.144850 Accuracy: 95.15% 4 Validation loss: 0.213363 Best loss: 0.144850 Accuracy: 94.57% 5 Validation loss: 0.261897 Best loss: 0.144850 Accuracy: 94.02% 6 Validation loss: 0.175300 Best loss: 0.144850 Accuracy: 95.97% 7 Validation loss: 0.221743 Best loss: 0.144850 Accuracy: 95.00% 8 Validation loss: 0.360451 Best loss: 0.144850 Accuracy: 89.87% 9 Validation loss: 0.215017 Best loss: 0.144850 Accuracy: 94.84% 10 Validation loss: 0.239491 Best loss: 0.144850 Accuracy: 93.78% 11 Validation loss: 0.330593 Best loss: 0.144850 Accuracy: 94.61% 12 Validation loss: 0.390891 Best loss: 0.144850 Accuracy: 90.54% 13 Validation loss: 0.613453 Best loss: 0.144850 Accuracy: 72.71% 14 Validation loss: 0.687957 Best loss: 0.144850 Accuracy: 69.66% 15 Validation loss: 0.676956 Best loss: 0.144850 Accuracy: 68.92% 16 Validation loss: 0.636507 Best loss: 0.144850 Accuracy: 71.46% 17 Validation loss: 0.507737 Best loss: 0.144850 Accuracy: 78.46% 18 Validation loss: 0.474475 Best loss: 0.144850 Accuracy: 86.86% 19 Validation loss: 0.342993 Best loss: 0.144850 Accuracy: 90.58% 20 Validation loss: 0.264914 Best loss: 0.144850 Accuracy: 93.94% 21 Validation loss: 0.622448 Best loss: 0.144850 Accuracy: 74.90% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 7.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02 0 Validation loss: 0.116745 Best loss: 0.116745 Accuracy: 96.60% 1 Validation loss: 0.083075 Best loss: 0.083075 Accuracy: 97.85% 2 Validation loss: 0.081494 Best loss: 0.081494 Accuracy: 97.69% 3 Validation loss: 0.069073 Best loss: 0.069073 Accuracy: 97.77% 4 Validation loss: 0.065523 Best loss: 0.065523 Accuracy: 97.93% 5 Validation loss: 0.064152 Best loss: 0.064152 Accuracy: 98.08% 6 Validation loss: 0.057036 Best loss: 0.057036 Accuracy: 98.16% 7 Validation loss: 0.049765 Best loss: 0.049765 Accuracy: 98.40% 8 Validation loss: 0.073306 Best loss: 0.049765 Accuracy: 97.81% 9 Validation loss: 0.054173 Best loss: 0.049765 Accuracy: 98.44% 10 Validation loss: 0.063803 Best loss: 0.049765 Accuracy: 98.36% 11 Validation loss: 0.072675 Best loss: 0.049765 Accuracy: 98.24% 12 Validation loss: 0.054565 Best loss: 0.049765 Accuracy: 98.63% 13 Validation loss: 0.059498 Best loss: 0.049765 Accuracy: 98.48% 14 Validation loss: 0.070949 Best loss: 0.049765 Accuracy: 98.20% 15 Validation loss: 0.073241 Best loss: 0.049765 Accuracy: 98.40% 16 Validation loss: 0.069066 Best loss: 0.049765 Accuracy: 98.40% 17 Validation loss: 0.068249 Best loss: 0.049765 Accuracy: 98.63% 18 Validation loss: 0.075653 Best loss: 0.049765 Accuracy: 98.55% 19 Validation loss: 0.084363 Best loss: 0.049765 Accuracy: 98.51% 20 Validation loss: 0.076573 Best loss: 0.049765 Accuracy: 98.59% 21 Validation loss: 0.090218 Best loss: 0.049765 Accuracy: 97.97% 22 Validation loss: 0.081998 Best loss: 0.049765 Accuracy: 98.59% 23 Validation loss: 0.095077 Best loss: 0.049765 Accuracy: 98.40% 24 Validation loss: 0.090252 Best loss: 0.049765 Accuracy: 98.63% 25 Validation loss: 0.107411 Best loss: 0.049765 Accuracy: 98.32% 26 Validation loss: 0.104350 Best loss: 0.049765 Accuracy: 98.40% 27 Validation loss: 0.083209 Best loss: 0.049765 Accuracy: 98.32% 28 Validation loss: 0.073992 Best loss: 0.049765 Accuracy: 98.59% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02, total= 2.6s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02 0 Validation loss: 0.116014 Best loss: 0.116014 Accuracy: 96.52% 1 Validation loss: 0.080407 Best loss: 0.080407 Accuracy: 97.58% 2 Validation loss: 0.069216 Best loss: 0.069216 Accuracy: 97.77% 3 Validation loss: 0.067354 Best loss: 0.067354 Accuracy: 97.77% 4 Validation loss: 0.080894 Best loss: 0.067354 Accuracy: 97.62% 5 Validation loss: 0.086315 Best loss: 0.067354 Accuracy: 97.65% 6 Validation loss: 0.073166 Best loss: 0.067354 Accuracy: 98.32% 7 Validation loss: 0.057299 Best loss: 0.057299 Accuracy: 98.51% 8 Validation loss: 0.058586 Best loss: 0.057299 Accuracy: 98.40% 9 Validation loss: 0.080324 Best loss: 0.057299 Accuracy: 98.08% 10 Validation loss: 0.071646 Best loss: 0.057299 Accuracy: 98.28% 11 Validation loss: 0.089936 Best loss: 0.057299 Accuracy: 98.28% 12 Validation loss: 0.075337 Best loss: 0.057299 Accuracy: 98.16% 13 Validation loss: 0.104162 Best loss: 0.057299 Accuracy: 97.54% 14 Validation loss: 0.085058 Best loss: 0.057299 Accuracy: 98.08% 15 Validation loss: 0.071897 Best loss: 0.057299 Accuracy: 98.36% 16 Validation loss: 0.095107 Best loss: 0.057299 Accuracy: 98.05% 17 Validation loss: 0.073366 Best loss: 0.057299 Accuracy: 98.28% 18 Validation loss: 0.094276 Best loss: 0.057299 Accuracy: 98.44% 19 Validation loss: 0.067479 Best loss: 0.057299 Accuracy: 98.51% 20 Validation loss: 0.087160 Best loss: 0.057299 Accuracy: 98.28% 21 Validation loss: 0.077615 Best loss: 0.057299 Accuracy: 98.28% 22 Validation loss: 0.084535 Best loss: 0.057299 Accuracy: 98.28% 23 Validation loss: 0.066662 Best loss: 0.057299 Accuracy: 98.79% 24 Validation loss: 0.086352 Best loss: 0.057299 Accuracy: 98.28% 25 Validation loss: 0.111996 Best loss: 0.057299 Accuracy: 98.05% 26 Validation loss: 0.093496 Best loss: 0.057299 Accuracy: 98.48% 27 Validation loss: 0.085869 Best loss: 0.057299 Accuracy: 98.44% 28 Validation loss: 0.081792 Best loss: 0.057299 Accuracy: 98.59% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02, total= 2.8s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02 0 Validation loss: 0.130424 Best loss: 0.130424 Accuracy: 96.05% 1 Validation loss: 0.088467 Best loss: 0.088467 Accuracy: 97.58% 2 Validation loss: 0.080924 Best loss: 0.080924 Accuracy: 97.46% 3 Validation loss: 0.067303 Best loss: 0.067303 Accuracy: 98.36% 4 Validation loss: 0.062449 Best loss: 0.062449 Accuracy: 98.20% 5 Validation loss: 0.066660 Best loss: 0.062449 Accuracy: 97.85% 6 Validation loss: 0.063659 Best loss: 0.062449 Accuracy: 98.05% 7 Validation loss: 0.065072 Best loss: 0.062449 Accuracy: 97.89% 8 Validation loss: 0.074696 Best loss: 0.062449 Accuracy: 98.08% 9 Validation loss: 0.067042 Best loss: 0.062449 Accuracy: 98.51% 10 Validation loss: 0.069242 Best loss: 0.062449 Accuracy: 98.24% 11 Validation loss: 0.079013 Best loss: 0.062449 Accuracy: 98.48% 12 Validation loss: 0.069915 Best loss: 0.062449 Accuracy: 98.75% 13 Validation loss: 0.152470 Best loss: 0.062449 Accuracy: 98.05% 14 Validation loss: 0.088409 Best loss: 0.062449 Accuracy: 98.01% 15 Validation loss: 0.083567 Best loss: 0.062449 Accuracy: 98.44% 16 Validation loss: 0.078005 Best loss: 0.062449 Accuracy: 98.48% 17 Validation loss: 0.088683 Best loss: 0.062449 Accuracy: 98.36% 18 Validation loss: 0.077004 Best loss: 0.062449 Accuracy: 98.48% 19 Validation loss: 0.072912 Best loss: 0.062449 Accuracy: 98.48% 20 Validation loss: 0.099815 Best loss: 0.062449 Accuracy: 98.44% 21 Validation loss: 0.082009 Best loss: 0.062449 Accuracy: 98.36% 22 Validation loss: 0.076940 Best loss: 0.062449 Accuracy: 98.59% 23 Validation loss: 0.070097 Best loss: 0.062449 Accuracy: 98.75% 24 Validation loss: 0.092534 Best loss: 0.062449 Accuracy: 98.48% 25 Validation loss: 0.164625 Best loss: 0.062449 Accuracy: 97.73% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=500, learning_rate=0.02, total= 2.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05 0 Validation loss: 2.309790 Best loss: 2.309790 Accuracy: 65.79% 1 Validation loss: 0.892112 Best loss: 0.892112 Accuracy: 69.27% 2 Validation loss: 0.543244 Best loss: 0.543244 Accuracy: 82.64% 3 Validation loss: 0.418952 Best loss: 0.418952 Accuracy: 88.27% 4 Validation loss: 0.409537 Best loss: 0.409537 Accuracy: 87.14% 5 Validation loss: 0.370640 Best loss: 0.370640 Accuracy: 92.10% 6 Validation loss: 0.213558 Best loss: 0.213558 Accuracy: 94.45% 7 Validation loss: 16013.588867 Best loss: 0.213558 Accuracy: 19.04% 8 Validation loss: 459.542328 Best loss: 0.213558 Accuracy: 35.89% 9 Validation loss: 129.630905 Best loss: 0.213558 Accuracy: 50.20% 10 Validation loss: 58.161125 Best loss: 0.213558 Accuracy: 63.76% 11 Validation loss: 15.776971 Best loss: 0.213558 Accuracy: 84.60% 12 Validation loss: 9.678064 Best loss: 0.213558 Accuracy: 87.72% 13 Validation loss: 74.114891 Best loss: 0.213558 Accuracy: 64.46% 14 Validation loss: 18.950293 Best loss: 0.213558 Accuracy: 70.17% 15 Validation loss: 7.993595 Best loss: 0.213558 Accuracy: 88.94% 16 Validation loss: 22.571108 Best loss: 0.213558 Accuracy: 75.57% 17 Validation loss: 7.664078 Best loss: 0.213558 Accuracy: 93.35% 18 Validation loss: 6.608495 Best loss: 0.213558 Accuracy: 94.68% 19 Validation loss: 9.695833 Best loss: 0.213558 Accuracy: 91.05% 20 Validation loss: 7.125737 Best loss: 0.213558 Accuracy: 95.04% 21 Validation loss: 4.325990 Best loss: 0.213558 Accuracy: 96.25% 22 Validation loss: 987.658997 Best loss: 0.213558 Accuracy: 58.21% 23 Validation loss: 186.292145 Best loss: 0.213558 Accuracy: 88.04% 24 Validation loss: 170.046814 Best loss: 0.213558 Accuracy: 87.65% 25 Validation loss: 638.975403 Best loss: 0.213558 Accuracy: 72.24% 26 Validation loss: 222.067856 Best loss: 0.213558 Accuracy: 87.49% 27 Validation loss: 176.842773 Best loss: 0.213558 Accuracy: 89.64% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05, total= 15.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05 0 Validation loss: 161.268417 Best loss: 161.268417 Accuracy: 20.91% 1 Validation loss: 2.417016 Best loss: 2.417016 Accuracy: 68.49% 2 Validation loss: 1.879838 Best loss: 1.879838 Accuracy: 69.66% 3 Validation loss: 2.710433 Best loss: 1.879838 Accuracy: 63.06% 4 Validation loss: 888.664673 Best loss: 1.879838 Accuracy: 38.86% 5 Validation loss: 134.384644 Best loss: 1.879838 Accuracy: 52.62% 6 Validation loss: 72.943420 Best loss: 1.879838 Accuracy: 67.63% 7 Validation loss: 58.607651 Best loss: 1.879838 Accuracy: 60.52% 8 Validation loss: 30.949179 Best loss: 1.879838 Accuracy: 72.13% 9 Validation loss: 24.902025 Best loss: 1.879838 Accuracy: 77.80% 10 Validation loss: 34.980400 Best loss: 1.879838 Accuracy: 72.09% 11 Validation loss: 590.146790 Best loss: 1.879838 Accuracy: 49.88% 12 Validation loss: 75.472862 Best loss: 1.879838 Accuracy: 63.64% 13 Validation loss: 581.136292 Best loss: 1.879838 Accuracy: 58.52% 14 Validation loss: 478.003662 Best loss: 1.879838 Accuracy: 56.76% 15 Validation loss: 108.651436 Best loss: 1.879838 Accuracy: 66.30% 16 Validation loss: 84.795303 Best loss: 1.879838 Accuracy: 76.66% 17 Validation loss: 59.257355 Best loss: 1.879838 Accuracy: 72.71% 18 Validation loss: 40747.160156 Best loss: 1.879838 Accuracy: 19.08% 19 Validation loss: 1704.500244 Best loss: 1.879838 Accuracy: 41.01% 20 Validation loss: 259.154175 Best loss: 1.879838 Accuracy: 72.99% 21 Validation loss: 184.387314 Best loss: 1.879838 Accuracy: 65.64% 22 Validation loss: 319.952698 Best loss: 1.879838 Accuracy: 62.20% 23 Validation loss: 83.806389 Best loss: 1.879838 Accuracy: 74.00% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05, total= 13.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05 0 Validation loss: 547.724304 Best loss: 547.724304 Accuracy: 22.01% 1 Validation loss: 8.091904 Best loss: 8.091904 Accuracy: 38.19% 2 Validation loss: 1.479471 Best loss: 1.479471 Accuracy: 43.59% 3 Validation loss: 1.014230 Best loss: 1.014230 Accuracy: 57.62% 4 Validation loss: 1.569293 Best loss: 1.014230 Accuracy: 51.88% 5 Validation loss: 0.881918 Best loss: 0.881918 Accuracy: 64.35% 6 Validation loss: 0.730612 Best loss: 0.730612 Accuracy: 69.74% 7 Validation loss: 0.889257 Best loss: 0.730612 Accuracy: 66.65% 8 Validation loss: 1.426594 Best loss: 0.730612 Accuracy: 66.93% 9 Validation loss: 0.440080 Best loss: 0.440080 Accuracy: 85.11% 10 Validation loss: 0.449887 Best loss: 0.440080 Accuracy: 88.12% 11 Validation loss: 39515.617188 Best loss: 0.440080 Accuracy: 11.73% 12 Validation loss: 195.377869 Best loss: 0.440080 Accuracy: 34.79% 13 Validation loss: 215.899628 Best loss: 0.440080 Accuracy: 34.17% 14 Validation loss: 133.725845 Best loss: 0.440080 Accuracy: 43.12% 15 Validation loss: 70.247299 Best loss: 0.440080 Accuracy: 58.48% 16 Validation loss: 61.179344 Best loss: 0.440080 Accuracy: 56.72% 17 Validation loss: 30.637037 Best loss: 0.440080 Accuracy: 72.67% 18 Validation loss: 32.667671 Best loss: 0.440080 Accuracy: 81.00% 19 Validation loss: 24.563663 Best loss: 0.440080 Accuracy: 78.54% 20 Validation loss: 21.890728 Best loss: 0.440080 Accuracy: 82.56% 21 Validation loss: 52.803062 Best loss: 0.440080 Accuracy: 74.35% 22 Validation loss: 12.013992 Best loss: 0.440080 Accuracy: 88.66% 23 Validation loss: 18.425812 Best loss: 0.440080 Accuracy: 84.75% 24 Validation loss: 14.824622 Best loss: 0.440080 Accuracy: 86.24% 25 Validation loss: 18.207891 Best loss: 0.440080 Accuracy: 82.33% 26 Validation loss: 12.449056 Best loss: 0.440080 Accuracy: 90.34% 27 Validation loss: 20.748503 Best loss: 0.440080 Accuracy: 90.42% 28 Validation loss: 19.644018 Best loss: 0.440080 Accuracy: 83.46% 29 Validation loss: 6.626858 Best loss: 0.440080 Accuracy: 91.32% 30 Validation loss: 4.680807 Best loss: 0.440080 Accuracy: 92.85% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=50, learning_rate=0.05, total= 17.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 14.778951 Best loss: 14.778951 Accuracy: 80.26% 1 Validation loss: 7.365212 Best loss: 7.365212 Accuracy: 77.76% 2 Validation loss: 3.960042 Best loss: 3.960042 Accuracy: 90.81% 3 Validation loss: 0.807410 Best loss: 0.807410 Accuracy: 93.59% 4 Validation loss: 3.353692 Best loss: 0.807410 Accuracy: 90.85% 5 Validation loss: 0.665015 Best loss: 0.665015 Accuracy: 95.58% 6 Validation loss: 0.906262 Best loss: 0.665015 Accuracy: 94.45% 7 Validation loss: 0.731274 Best loss: 0.665015 Accuracy: 91.75% 8 Validation loss: 0.486407 Best loss: 0.486407 Accuracy: 95.50% 9 Validation loss: 0.517267 Best loss: 0.486407 Accuracy: 93.98% 10 Validation loss: 0.362084 Best loss: 0.362084 Accuracy: 95.82% 11 Validation loss: 159932.500000 Best loss: 0.362084 Accuracy: 81.90% 12 Validation loss: 125742.562500 Best loss: 0.362084 Accuracy: 68.37% 13 Validation loss: 50368.472656 Best loss: 0.362084 Accuracy: 79.05% 14 Validation loss: 10966.552734 Best loss: 0.362084 Accuracy: 94.14% 15 Validation loss: 14481.068359 Best loss: 0.362084 Accuracy: 94.57% 16 Validation loss: 12213.264648 Best loss: 0.362084 Accuracy: 95.07% 17 Validation loss: 11498.218750 Best loss: 0.362084 Accuracy: 93.71% 18 Validation loss: 10797.661133 Best loss: 0.362084 Accuracy: 93.98% 19 Validation loss: 16932.236328 Best loss: 0.362084 Accuracy: 87.14% 20 Validation loss: 19205.242188 Best loss: 0.362084 Accuracy: 93.63% 21 Validation loss: 5431.210938 Best loss: 0.362084 Accuracy: 95.97% 22 Validation loss: 25177.138672 Best loss: 0.362084 Accuracy: 79.91% 23 Validation loss: 11187.376953 Best loss: 0.362084 Accuracy: 95.54% 24 Validation loss: 5144.792969 Best loss: 0.362084 Accuracy: 96.56% 25 Validation loss: 6216.220703 Best loss: 0.362084 Accuracy: 95.50% 26 Validation loss: 4888.697754 Best loss: 0.362084 Accuracy: 95.47% 27 Validation loss: 10681.485352 Best loss: 0.362084 Accuracy: 95.43% 28 Validation loss: 2500.686768 Best loss: 0.362084 Accuracy: 96.44% 29 Validation loss: 3103.936523 Best loss: 0.362084 Accuracy: 92.65% 30 Validation loss: 1526.866821 Best loss: 0.362084 Accuracy: 96.79% 31 Validation loss: 6986231.500000 Best loss: 0.362084 Accuracy: 56.29% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 26.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 0.296677 Best loss: 0.296677 Accuracy: 92.96% 1 Validation loss: 0.204553 Best loss: 0.204553 Accuracy: 95.43% 2 Validation loss: 0.134779 Best loss: 0.134779 Accuracy: 95.93% 3 Validation loss: 13479.458984 Best loss: 0.134779 Accuracy: 87.37% 4 Validation loss: 2140.639648 Best loss: 0.134779 Accuracy: 93.63% 5 Validation loss: 1756.766602 Best loss: 0.134779 Accuracy: 94.14% 6 Validation loss: 808.621155 Best loss: 0.134779 Accuracy: 95.86% 7 Validation loss: 1150.520386 Best loss: 0.134779 Accuracy: 94.33% 8 Validation loss: 1608.799927 Best loss: 0.134779 Accuracy: 90.27% 9 Validation loss: 768.749756 Best loss: 0.134779 Accuracy: 95.07% 10 Validation loss: 749.278564 Best loss: 0.134779 Accuracy: 95.78% 11 Validation loss: 564.982727 Best loss: 0.134779 Accuracy: 95.31% 12 Validation loss: 618.527710 Best loss: 0.134779 Accuracy: 94.92% 13 Validation loss: 9536.251953 Best loss: 0.134779 Accuracy: 89.60% 14 Validation loss: 705.616211 Best loss: 0.134779 Accuracy: 94.76% 15 Validation loss: 625.111938 Best loss: 0.134779 Accuracy: 94.45% 16 Validation loss: 552.118835 Best loss: 0.134779 Accuracy: 95.19% 17 Validation loss: 339.960205 Best loss: 0.134779 Accuracy: 96.13% 18 Validation loss: 242.085785 Best loss: 0.134779 Accuracy: 96.13% 19 Validation loss: 210292.796875 Best loss: 0.134779 Accuracy: 87.45% 20 Validation loss: 94681.539062 Best loss: 0.134779 Accuracy: 88.98% 21 Validation loss: 166810.671875 Best loss: 0.134779 Accuracy: 77.29% 22 Validation loss: 77329.476562 Best loss: 0.134779 Accuracy: 90.77% 23 Validation loss: 51177.101562 Best loss: 0.134779 Accuracy: 92.96% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 23.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 127.526703 Best loss: 127.526703 Accuracy: 64.74% 1 Validation loss: 15.327091 Best loss: 15.327091 Accuracy: 85.42% 2 Validation loss: 16.758020 Best loss: 15.327091 Accuracy: 76.47% 3 Validation loss: 5.218018 Best loss: 5.218018 Accuracy: 91.91% 4 Validation loss: 10.006907 Best loss: 5.218018 Accuracy: 91.91% 5 Validation loss: 7.554106 Best loss: 5.218018 Accuracy: 92.53% 6 Validation loss: 3.109525 Best loss: 3.109525 Accuracy: 95.47% 7 Validation loss: 4.849286 Best loss: 3.109525 Accuracy: 90.58% 8 Validation loss: 2.329339 Best loss: 2.329339 Accuracy: 95.39% 9 Validation loss: 97287.617188 Best loss: 2.329339 Accuracy: 76.82% 10 Validation loss: 45950.222656 Best loss: 2.329339 Accuracy: 84.52% 11 Validation loss: 62249.902344 Best loss: 2.329339 Accuracy: 81.67% 12 Validation loss: 42291.195312 Best loss: 2.329339 Accuracy: 87.22% 13 Validation loss: 21411.402344 Best loss: 2.329339 Accuracy: 93.59% 14 Validation loss: 30552.544922 Best loss: 2.329339 Accuracy: 90.23% 15 Validation loss: 27199.656250 Best loss: 2.329339 Accuracy: 90.30% 16 Validation loss: 17806.517578 Best loss: 2.329339 Accuracy: 91.24% 17 Validation loss: 21985.373047 Best loss: 2.329339 Accuracy: 90.70% 18 Validation loss: 28959.509766 Best loss: 2.329339 Accuracy: 90.46% 19 Validation loss: 18184.263672 Best loss: 2.329339 Accuracy: 88.66% 20 Validation loss: 9300.428711 Best loss: 2.329339 Accuracy: 95.11% 21 Validation loss: 2524330.250000 Best loss: 2.329339 Accuracy: 87.45% 22 Validation loss: 167252.046875 Best loss: 2.329339 Accuracy: 92.34% 23 Validation loss: 35623.695312 Best loss: 2.329339 Accuracy: 96.60% 24 Validation loss: 45604.296875 Best loss: 2.329339 Accuracy: 95.47% 25 Validation loss: 26380.005859 Best loss: 2.329339 Accuracy: 95.82% 26 Validation loss: 21112.732422 Best loss: 2.329339 Accuracy: 96.05% 27 Validation loss: 20601.589844 Best loss: 2.329339 Accuracy: 96.40% 28 Validation loss: 71421.476562 Best loss: 2.329339 Accuracy: 90.81% 29 Validation loss: 19197.861328 Best loss: 2.329339 Accuracy: 94.06% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 29.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01 0 Validation loss: 0.091549 Best loss: 0.091549 Accuracy: 97.22% 1 Validation loss: 0.065673 Best loss: 0.065673 Accuracy: 97.81% 2 Validation loss: 0.057730 Best loss: 0.057730 Accuracy: 98.28% 3 Validation loss: 0.048221 Best loss: 0.048221 Accuracy: 98.44% 4 Validation loss: 0.052076 Best loss: 0.048221 Accuracy: 98.48% 5 Validation loss: 0.065433 Best loss: 0.048221 Accuracy: 98.40% 6 Validation loss: 0.060498 Best loss: 0.048221 Accuracy: 98.48% 7 Validation loss: 0.064470 Best loss: 0.048221 Accuracy: 98.67% 8 Validation loss: 0.070457 Best loss: 0.048221 Accuracy: 98.32% 9 Validation loss: 0.071641 Best loss: 0.048221 Accuracy: 98.87% 10 Validation loss: 0.071373 Best loss: 0.048221 Accuracy: 98.79% 11 Validation loss: 0.086256 Best loss: 0.048221 Accuracy: 98.67% 12 Validation loss: 0.065449 Best loss: 0.048221 Accuracy: 98.71% 13 Validation loss: 0.062018 Best loss: 0.048221 Accuracy: 98.79% 14 Validation loss: 0.054089 Best loss: 0.048221 Accuracy: 98.83% 15 Validation loss: 0.078551 Best loss: 0.048221 Accuracy: 98.48% 16 Validation loss: 0.073429 Best loss: 0.048221 Accuracy: 98.94% 17 Validation loss: 0.080221 Best loss: 0.048221 Accuracy: 99.02% 18 Validation loss: 0.067301 Best loss: 0.048221 Accuracy: 98.71% 19 Validation loss: 0.063883 Best loss: 0.048221 Accuracy: 98.94% 20 Validation loss: 0.058665 Best loss: 0.048221 Accuracy: 98.79% 21 Validation loss: 0.073396 Best loss: 0.048221 Accuracy: 98.75% 22 Validation loss: 0.064099 Best loss: 0.048221 Accuracy: 98.83% 23 Validation loss: 0.076498 Best loss: 0.048221 Accuracy: 98.71% 24 Validation loss: 0.082193 Best loss: 0.048221 Accuracy: 98.40% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01, total= 4.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01 0 Validation loss: 0.100772 Best loss: 0.100772 Accuracy: 97.26% 1 Validation loss: 0.061683 Best loss: 0.061683 Accuracy: 98.28% 2 Validation loss: 0.046005 Best loss: 0.046005 Accuracy: 98.48% 3 Validation loss: 0.070749 Best loss: 0.046005 Accuracy: 98.05% 4 Validation loss: 0.053079 Best loss: 0.046005 Accuracy: 98.55% 5 Validation loss: 0.064340 Best loss: 0.046005 Accuracy: 98.59% 6 Validation loss: 0.055064 Best loss: 0.046005 Accuracy: 98.40% 7 Validation loss: 0.050467 Best loss: 0.046005 Accuracy: 98.59% 8 Validation loss: 0.054381 Best loss: 0.046005 Accuracy: 98.79% 9 Validation loss: 0.046008 Best loss: 0.046005 Accuracy: 98.79% 10 Validation loss: 0.043847 Best loss: 0.043847 Accuracy: 98.91% 11 Validation loss: 0.049128 Best loss: 0.043847 Accuracy: 98.94% 12 Validation loss: 0.037903 Best loss: 0.037903 Accuracy: 98.94% 13 Validation loss: 0.050760 Best loss: 0.037903 Accuracy: 98.71% 14 Validation loss: 0.057986 Best loss: 0.037903 Accuracy: 98.91% 15 Validation loss: 0.066152 Best loss: 0.037903 Accuracy: 98.63% 16 Validation loss: 0.051118 Best loss: 0.037903 Accuracy: 98.94% 17 Validation loss: 0.056521 Best loss: 0.037903 Accuracy: 98.71% 18 Validation loss: 0.045851 Best loss: 0.037903 Accuracy: 99.02% 19 Validation loss: 0.054751 Best loss: 0.037903 Accuracy: 98.91% 20 Validation loss: 0.047885 Best loss: 0.037903 Accuracy: 99.10% 21 Validation loss: 0.055340 Best loss: 0.037903 Accuracy: 99.02% 22 Validation loss: 0.114922 Best loss: 0.037903 Accuracy: 98.20% 23 Validation loss: 0.207607 Best loss: 0.037903 Accuracy: 94.45% 24 Validation loss: 0.137386 Best loss: 0.037903 Accuracy: 96.79% 25 Validation loss: 0.080408 Best loss: 0.037903 Accuracy: 97.69% 26 Validation loss: 0.076374 Best loss: 0.037903 Accuracy: 98.16% 27 Validation loss: 0.071834 Best loss: 0.037903 Accuracy: 98.08% 28 Validation loss: 0.070221 Best loss: 0.037903 Accuracy: 98.40% 29 Validation loss: 0.067425 Best loss: 0.037903 Accuracy: 98.36% 30 Validation loss: 0.068379 Best loss: 0.037903 Accuracy: 98.55% 31 Validation loss: 0.070567 Best loss: 0.037903 Accuracy: 98.36% 32 Validation loss: 0.084383 Best loss: 0.037903 Accuracy: 98.40% 33 Validation loss: 0.078874 Best loss: 0.037903 Accuracy: 98.63% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01, total= 5.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01 0 Validation loss: 0.087257 Best loss: 0.087257 Accuracy: 97.15% 1 Validation loss: 0.064454 Best loss: 0.064454 Accuracy: 98.28% 2 Validation loss: 0.055231 Best loss: 0.055231 Accuracy: 98.05% 3 Validation loss: 0.060147 Best loss: 0.055231 Accuracy: 98.55% 4 Validation loss: 0.062698 Best loss: 0.055231 Accuracy: 98.48% 5 Validation loss: 0.050512 Best loss: 0.050512 Accuracy: 98.44% 6 Validation loss: 0.050065 Best loss: 0.050065 Accuracy: 98.71% 7 Validation loss: 0.059983 Best loss: 0.050065 Accuracy: 98.40% 8 Validation loss: 0.055281 Best loss: 0.050065 Accuracy: 98.48% 9 Validation loss: 0.070319 Best loss: 0.050065 Accuracy: 98.51% 10 Validation loss: 0.059696 Best loss: 0.050065 Accuracy: 98.67% 11 Validation loss: 0.072101 Best loss: 0.050065 Accuracy: 98.67% 12 Validation loss: 0.056733 Best loss: 0.050065 Accuracy: 98.55% 13 Validation loss: 0.066019 Best loss: 0.050065 Accuracy: 98.63% 14 Validation loss: 0.048932 Best loss: 0.048932 Accuracy: 98.87% 15 Validation loss: 0.090801 Best loss: 0.048932 Accuracy: 98.71% 16 Validation loss: 0.069932 Best loss: 0.048932 Accuracy: 98.67% 17 Validation loss: 0.082875 Best loss: 0.048932 Accuracy: 98.67% 18 Validation loss: 0.087946 Best loss: 0.048932 Accuracy: 98.75% 19 Validation loss: 0.061555 Best loss: 0.048932 Accuracy: 98.79% 20 Validation loss: 0.074979 Best loss: 0.048932 Accuracy: 98.79% 21 Validation loss: 0.059236 Best loss: 0.048932 Accuracy: 98.75% 22 Validation loss: 0.082801 Best loss: 0.048932 Accuracy: 98.67% 23 Validation loss: 0.155897 Best loss: 0.048932 Accuracy: 98.59% 24 Validation loss: 0.194036 Best loss: 0.048932 Accuracy: 96.56% 25 Validation loss: 0.327685 Best loss: 0.048932 Accuracy: 92.18% 26 Validation loss: 0.103769 Best loss: 0.048932 Accuracy: 97.58% 27 Validation loss: 0.075313 Best loss: 0.048932 Accuracy: 98.01% 28 Validation loss: 0.065192 Best loss: 0.048932 Accuracy: 98.28% 29 Validation loss: 0.066623 Best loss: 0.048932 Accuracy: 98.44% 30 Validation loss: 0.072080 Best loss: 0.048932 Accuracy: 98.36% 31 Validation loss: 0.064671 Best loss: 0.048932 Accuracy: 98.44% 32 Validation loss: 0.093815 Best loss: 0.048932 Accuracy: 98.55% 33 Validation loss: 0.088002 Best loss: 0.048932 Accuracy: 98.71% 34 Validation loss: 0.110193 Best loss: 0.048932 Accuracy: 98.51% 35 Validation loss: 0.093825 Best loss: 0.048932 Accuracy: 98.59% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=120, batch_size=500, learning_rate=0.01, total= 5.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01 0 Validation loss: 0.099634 Best loss: 0.099634 Accuracy: 96.87% 1 Validation loss: 0.060449 Best loss: 0.060449 Accuracy: 98.24% 2 Validation loss: 0.063522 Best loss: 0.060449 Accuracy: 98.16% 3 Validation loss: 0.056077 Best loss: 0.056077 Accuracy: 98.28% 4 Validation loss: 0.058080 Best loss: 0.056077 Accuracy: 98.16% 5 Validation loss: 0.051749 Best loss: 0.051749 Accuracy: 98.67% 6 Validation loss: 0.044574 Best loss: 0.044574 Accuracy: 98.98% 7 Validation loss: 0.090230 Best loss: 0.044574 Accuracy: 97.62% 8 Validation loss: 0.061477 Best loss: 0.044574 Accuracy: 98.67% 9 Validation loss: 0.078075 Best loss: 0.044574 Accuracy: 98.24% 10 Validation loss: 0.055543 Best loss: 0.044574 Accuracy: 98.83% 11 Validation loss: 0.058818 Best loss: 0.044574 Accuracy: 98.71% 12 Validation loss: 0.044955 Best loss: 0.044574 Accuracy: 99.18% 13 Validation loss: 0.067300 Best loss: 0.044574 Accuracy: 98.59% 14 Validation loss: 0.063940 Best loss: 0.044574 Accuracy: 98.83% 15 Validation loss: 0.068076 Best loss: 0.044574 Accuracy: 98.63% 16 Validation loss: 0.048091 Best loss: 0.044574 Accuracy: 98.87% 17 Validation loss: 0.072479 Best loss: 0.044574 Accuracy: 98.75% 18 Validation loss: 0.069975 Best loss: 0.044574 Accuracy: 98.51% 19 Validation loss: 0.073660 Best loss: 0.044574 Accuracy: 98.63% 20 Validation loss: 0.067799 Best loss: 0.044574 Accuracy: 98.71% 21 Validation loss: 0.097945 Best loss: 0.044574 Accuracy: 98.79% 22 Validation loss: 0.057800 Best loss: 0.044574 Accuracy: 98.91% 23 Validation loss: 0.059879 Best loss: 0.044574 Accuracy: 99.10% 24 Validation loss: 0.060674 Best loss: 0.044574 Accuracy: 98.75% 25 Validation loss: 0.077725 Best loss: 0.044574 Accuracy: 98.98% 26 Validation loss: 0.067117 Best loss: 0.044574 Accuracy: 99.18% 27 Validation loss: 0.071216 Best loss: 0.044574 Accuracy: 98.94% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01, total= 4.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01 0 Validation loss: 0.107705 Best loss: 0.107705 Accuracy: 96.91% 1 Validation loss: 0.068719 Best loss: 0.068719 Accuracy: 97.89% 2 Validation loss: 0.064557 Best loss: 0.064557 Accuracy: 97.81% 3 Validation loss: 0.047841 Best loss: 0.047841 Accuracy: 98.55% 4 Validation loss: 0.055193 Best loss: 0.047841 Accuracy: 98.24% 5 Validation loss: 0.051390 Best loss: 0.047841 Accuracy: 98.48% 6 Validation loss: 0.046553 Best loss: 0.046553 Accuracy: 98.59% 7 Validation loss: 0.048525 Best loss: 0.046553 Accuracy: 98.87% 8 Validation loss: 0.054840 Best loss: 0.046553 Accuracy: 98.67% 9 Validation loss: 0.066456 Best loss: 0.046553 Accuracy: 98.59% 10 Validation loss: 0.057222 Best loss: 0.046553 Accuracy: 98.98% 11 Validation loss: 0.073248 Best loss: 0.046553 Accuracy: 98.44% 12 Validation loss: 0.064334 Best loss: 0.046553 Accuracy: 98.75% 13 Validation loss: 0.053766 Best loss: 0.046553 Accuracy: 98.75% 14 Validation loss: 0.056901 Best loss: 0.046553 Accuracy: 98.59% 15 Validation loss: 0.044412 Best loss: 0.044412 Accuracy: 99.14% 16 Validation loss: 0.057306 Best loss: 0.044412 Accuracy: 98.87% 17 Validation loss: 0.060575 Best loss: 0.044412 Accuracy: 98.63% 18 Validation loss: 0.043678 Best loss: 0.043678 Accuracy: 98.91% 19 Validation loss: 0.061811 Best loss: 0.043678 Accuracy: 98.98% 20 Validation loss: 0.051677 Best loss: 0.043678 Accuracy: 98.83% 21 Validation loss: 0.057105 Best loss: 0.043678 Accuracy: 99.06% 22 Validation loss: 0.052398 Best loss: 0.043678 Accuracy: 98.91% 23 Validation loss: 0.059367 Best loss: 0.043678 Accuracy: 99.18% 24 Validation loss: 0.069751 Best loss: 0.043678 Accuracy: 99.06% 25 Validation loss: 0.063301 Best loss: 0.043678 Accuracy: 99.10% 26 Validation loss: 0.101344 Best loss: 0.043678 Accuracy: 98.63% 27 Validation loss: 0.069793 Best loss: 0.043678 Accuracy: 98.67% 28 Validation loss: 0.044817 Best loss: 0.043678 Accuracy: 99.06% 29 Validation loss: 0.056354 Best loss: 0.043678 Accuracy: 98.91% 30 Validation loss: 0.040224 Best loss: 0.040224 Accuracy: 99.10% 31 Validation loss: 0.052428 Best loss: 0.040224 Accuracy: 99.34% 32 Validation loss: 0.058688 Best loss: 0.040224 Accuracy: 99.14% 33 Validation loss: 0.050467 Best loss: 0.040224 Accuracy: 99.18% 34 Validation loss: 0.065972 Best loss: 0.040224 Accuracy: 99.10% 35 Validation loss: 0.077250 Best loss: 0.040224 Accuracy: 99.22% 36 Validation loss: 0.085930 Best loss: 0.040224 Accuracy: 99.26% 37 Validation loss: 0.079728 Best loss: 0.040224 Accuracy: 99.18% 38 Validation loss: 0.077883 Best loss: 0.040224 Accuracy: 99.14% 39 Validation loss: 0.080154 Best loss: 0.040224 Accuracy: 99.14% 40 Validation loss: 0.080874 Best loss: 0.040224 Accuracy: 99.14% 41 Validation loss: 0.081364 Best loss: 0.040224 Accuracy: 99.14% 42 Validation loss: 0.081946 Best loss: 0.040224 Accuracy: 99.14% 43 Validation loss: 0.082462 Best loss: 0.040224 Accuracy: 99.18% 44 Validation loss: 0.082860 Best loss: 0.040224 Accuracy: 99.18% 45 Validation loss: 0.083311 Best loss: 0.040224 Accuracy: 99.18% 46 Validation loss: 0.083749 Best loss: 0.040224 Accuracy: 99.18% 47 Validation loss: 0.084160 Best loss: 0.040224 Accuracy: 99.18% 48 Validation loss: 0.084564 Best loss: 0.040224 Accuracy: 99.18% 49 Validation loss: 0.084946 Best loss: 0.040224 Accuracy: 99.18% 50 Validation loss: 0.085304 Best loss: 0.040224 Accuracy: 99.18% 51 Validation loss: 0.085637 Best loss: 0.040224 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01, total= 7.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01 0 Validation loss: 0.099424 Best loss: 0.099424 Accuracy: 96.95% 1 Validation loss: 0.068889 Best loss: 0.068889 Accuracy: 98.24% 2 Validation loss: 0.060660 Best loss: 0.060660 Accuracy: 98.12% 3 Validation loss: 0.068985 Best loss: 0.060660 Accuracy: 98.08% 4 Validation loss: 0.055712 Best loss: 0.055712 Accuracy: 98.20% 5 Validation loss: 0.052873 Best loss: 0.052873 Accuracy: 98.44% 6 Validation loss: 0.045001 Best loss: 0.045001 Accuracy: 98.79% 7 Validation loss: 0.040732 Best loss: 0.040732 Accuracy: 98.94% 8 Validation loss: 0.069681 Best loss: 0.040732 Accuracy: 98.08% 9 Validation loss: 0.059990 Best loss: 0.040732 Accuracy: 98.71% 10 Validation loss: 0.047676 Best loss: 0.040732 Accuracy: 98.75% 11 Validation loss: 0.051472 Best loss: 0.040732 Accuracy: 98.94% 12 Validation loss: 0.063302 Best loss: 0.040732 Accuracy: 98.94% 13 Validation loss: 0.063630 Best loss: 0.040732 Accuracy: 98.59% 14 Validation loss: 0.052617 Best loss: 0.040732 Accuracy: 98.94% 15 Validation loss: 0.047805 Best loss: 0.040732 Accuracy: 98.83% 16 Validation loss: 0.065327 Best loss: 0.040732 Accuracy: 99.02% 17 Validation loss: 0.070564 Best loss: 0.040732 Accuracy: 98.91% 18 Validation loss: 0.078589 Best loss: 0.040732 Accuracy: 98.44% 19 Validation loss: 0.055916 Best loss: 0.040732 Accuracy: 98.63% 20 Validation loss: 0.049301 Best loss: 0.040732 Accuracy: 98.75% 21 Validation loss: 0.067433 Best loss: 0.040732 Accuracy: 98.71% 22 Validation loss: 0.064399 Best loss: 0.040732 Accuracy: 98.79% 23 Validation loss: 0.052254 Best loss: 0.040732 Accuracy: 98.63% 24 Validation loss: 0.063926 Best loss: 0.040732 Accuracy: 98.75% 25 Validation loss: 0.052884 Best loss: 0.040732 Accuracy: 98.87% 26 Validation loss: 0.062907 Best loss: 0.040732 Accuracy: 98.75% 27 Validation loss: 0.081728 Best loss: 0.040732 Accuracy: 98.63% 28 Validation loss: 0.063551 Best loss: 0.040732 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=90, batch_size=500, learning_rate=0.01, total= 4.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.126311 Best loss: 0.126311 Accuracy: 96.09% 1 Validation loss: 0.075660 Best loss: 0.075660 Accuracy: 97.26% 2 Validation loss: 0.065916 Best loss: 0.065916 Accuracy: 97.97% 3 Validation loss: 0.062634 Best loss: 0.062634 Accuracy: 98.05% 4 Validation loss: 0.062960 Best loss: 0.062634 Accuracy: 98.08% 5 Validation loss: 0.049125 Best loss: 0.049125 Accuracy: 98.40% 6 Validation loss: 0.068804 Best loss: 0.049125 Accuracy: 98.32% 7 Validation loss: 0.067783 Best loss: 0.049125 Accuracy: 98.20% 8 Validation loss: 0.070697 Best loss: 0.049125 Accuracy: 98.05% 9 Validation loss: 0.054311 Best loss: 0.049125 Accuracy: 98.75% 10 Validation loss: 0.062787 Best loss: 0.049125 Accuracy: 98.87% 11 Validation loss: 0.060841 Best loss: 0.049125 Accuracy: 98.75% 12 Validation loss: 0.055216 Best loss: 0.049125 Accuracy: 98.63% 13 Validation loss: 0.075793 Best loss: 0.049125 Accuracy: 98.55% 14 Validation loss: 0.055670 Best loss: 0.049125 Accuracy: 98.63% 15 Validation loss: 0.078605 Best loss: 0.049125 Accuracy: 98.51% 16 Validation loss: 0.051717 Best loss: 0.049125 Accuracy: 98.79% 17 Validation loss: 0.070303 Best loss: 0.049125 Accuracy: 98.75% 18 Validation loss: 0.069324 Best loss: 0.049125 Accuracy: 98.83% 19 Validation loss: 0.051422 Best loss: 0.049125 Accuracy: 98.87% 20 Validation loss: 0.081769 Best loss: 0.049125 Accuracy: 98.48% 21 Validation loss: 0.064386 Best loss: 0.049125 Accuracy: 98.67% 22 Validation loss: 0.049595 Best loss: 0.049125 Accuracy: 98.71% 23 Validation loss: 0.057198 Best loss: 0.049125 Accuracy: 98.87% 24 Validation loss: 0.065693 Best loss: 0.049125 Accuracy: 98.71% 25 Validation loss: 0.048917 Best loss: 0.048917 Accuracy: 98.98% 26 Validation loss: 0.054821 Best loss: 0.048917 Accuracy: 98.79% 27 Validation loss: 0.120277 Best loss: 0.048917 Accuracy: 98.48% 28 Validation loss: 0.102256 Best loss: 0.048917 Accuracy: 98.87% 29 Validation loss: 38.947002 Best loss: 0.048917 Accuracy: 48.01% 30 Validation loss: 1.645353 Best loss: 0.048917 Accuracy: 20.91% 31 Validation loss: 1.623044 Best loss: 0.048917 Accuracy: 22.01% 32 Validation loss: 1.621862 Best loss: 0.048917 Accuracy: 22.01% 33 Validation loss: 1.619291 Best loss: 0.048917 Accuracy: 19.27% 34 Validation loss: 1.616686 Best loss: 0.048917 Accuracy: 19.08% 35 Validation loss: 1.634146 Best loss: 0.048917 Accuracy: 19.27% 36 Validation loss: 1.616428 Best loss: 0.048917 Accuracy: 20.91% 37 Validation loss: 1.622200 Best loss: 0.048917 Accuracy: 22.01% 38 Validation loss: 1.612510 Best loss: 0.048917 Accuracy: 22.01% 39 Validation loss: 1.608702 Best loss: 0.048917 Accuracy: 22.01% 40 Validation loss: 1.619697 Best loss: 0.048917 Accuracy: 20.91% 41 Validation loss: 1.628078 Best loss: 0.048917 Accuracy: 19.27% 42 Validation loss: 1.619168 Best loss: 0.048917 Accuracy: 22.01% 43 Validation loss: 1.627289 Best loss: 0.048917 Accuracy: 18.73% 44 Validation loss: 1.631504 Best loss: 0.048917 Accuracy: 19.08% 45 Validation loss: 1.613353 Best loss: 0.048917 Accuracy: 22.01% 46 Validation loss: 1.618325 Best loss: 0.048917 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 5.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.115145 Best loss: 0.115145 Accuracy: 96.56% 1 Validation loss: 0.074679 Best loss: 0.074679 Accuracy: 97.81% 2 Validation loss: 0.059059 Best loss: 0.059059 Accuracy: 97.97% 3 Validation loss: 0.062662 Best loss: 0.059059 Accuracy: 98.24% 4 Validation loss: 0.054455 Best loss: 0.054455 Accuracy: 98.48% 5 Validation loss: 0.052157 Best loss: 0.052157 Accuracy: 98.28% 6 Validation loss: 0.053113 Best loss: 0.052157 Accuracy: 98.36% 7 Validation loss: 0.042578 Best loss: 0.042578 Accuracy: 98.67% 8 Validation loss: 0.054133 Best loss: 0.042578 Accuracy: 98.63% 9 Validation loss: 0.045164 Best loss: 0.042578 Accuracy: 98.55% 10 Validation loss: 0.076807 Best loss: 0.042578 Accuracy: 98.59% 11 Validation loss: 0.060775 Best loss: 0.042578 Accuracy: 98.71% 12 Validation loss: 0.058802 Best loss: 0.042578 Accuracy: 98.67% 13 Validation loss: 0.048951 Best loss: 0.042578 Accuracy: 98.98% 14 Validation loss: 0.067934 Best loss: 0.042578 Accuracy: 98.94% 15 Validation loss: 0.064123 Best loss: 0.042578 Accuracy: 98.55% 16 Validation loss: 0.095762 Best loss: 0.042578 Accuracy: 98.24% 17 Validation loss: 0.079538 Best loss: 0.042578 Accuracy: 98.59% 18 Validation loss: 0.060185 Best loss: 0.042578 Accuracy: 98.79% 19 Validation loss: 0.041433 Best loss: 0.041433 Accuracy: 99.02% 20 Validation loss: 0.048227 Best loss: 0.041433 Accuracy: 99.06% 21 Validation loss: 0.055892 Best loss: 0.041433 Accuracy: 98.83% 22 Validation loss: 0.056000 Best loss: 0.041433 Accuracy: 98.91% 23 Validation loss: 0.067924 Best loss: 0.041433 Accuracy: 98.63% 24 Validation loss: 0.045604 Best loss: 0.041433 Accuracy: 98.87% 25 Validation loss: 0.054557 Best loss: 0.041433 Accuracy: 98.83% 26 Validation loss: 0.058597 Best loss: 0.041433 Accuracy: 98.91% 27 Validation loss: 0.061782 Best loss: 0.041433 Accuracy: 99.02% 28 Validation loss: 0.054513 Best loss: 0.041433 Accuracy: 98.98% 29 Validation loss: 0.086430 Best loss: 0.041433 Accuracy: 98.71% 30 Validation loss: 0.071247 Best loss: 0.041433 Accuracy: 98.40% 31 Validation loss: 0.054283 Best loss: 0.041433 Accuracy: 98.79% 32 Validation loss: 0.055135 Best loss: 0.041433 Accuracy: 98.91% 33 Validation loss: 0.080311 Best loss: 0.041433 Accuracy: 98.63% 34 Validation loss: 0.075549 Best loss: 0.041433 Accuracy: 98.87% 35 Validation loss: 0.073870 Best loss: 0.041433 Accuracy: 98.98% 36 Validation loss: 0.052869 Best loss: 0.041433 Accuracy: 98.87% 37 Validation loss: 0.054665 Best loss: 0.041433 Accuracy: 98.94% 38 Validation loss: 0.065968 Best loss: 0.041433 Accuracy: 98.67% 39 Validation loss: 0.074331 Best loss: 0.041433 Accuracy: 98.71% 40 Validation loss: 0.074977 Best loss: 0.041433 Accuracy: 98.79% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 5.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.116423 Best loss: 0.116423 Accuracy: 96.48% 1 Validation loss: 0.081041 Best loss: 0.081041 Accuracy: 97.62% 2 Validation loss: 0.075209 Best loss: 0.075209 Accuracy: 97.58% 3 Validation loss: 0.056255 Best loss: 0.056255 Accuracy: 98.28% 4 Validation loss: 0.049450 Best loss: 0.049450 Accuracy: 98.28% 5 Validation loss: 0.054106 Best loss: 0.049450 Accuracy: 98.28% 6 Validation loss: 0.070078 Best loss: 0.049450 Accuracy: 97.85% 7 Validation loss: 0.047719 Best loss: 0.047719 Accuracy: 98.55% 8 Validation loss: 0.059102 Best loss: 0.047719 Accuracy: 98.51% 9 Validation loss: 0.055899 Best loss: 0.047719 Accuracy: 98.40% 10 Validation loss: 0.072127 Best loss: 0.047719 Accuracy: 98.24% 11 Validation loss: 0.053615 Best loss: 0.047719 Accuracy: 98.55% 12 Validation loss: 0.061575 Best loss: 0.047719 Accuracy: 98.51% 13 Validation loss: 0.060236 Best loss: 0.047719 Accuracy: 98.59% 14 Validation loss: 0.053571 Best loss: 0.047719 Accuracy: 98.71% 15 Validation loss: 0.053652 Best loss: 0.047719 Accuracy: 98.94% 16 Validation loss: 0.074678 Best loss: 0.047719 Accuracy: 98.59% 17 Validation loss: 0.064904 Best loss: 0.047719 Accuracy: 98.63% 18 Validation loss: 0.067983 Best loss: 0.047719 Accuracy: 98.51% 19 Validation loss: 0.054842 Best loss: 0.047719 Accuracy: 98.94% 20 Validation loss: 0.049299 Best loss: 0.047719 Accuracy: 98.91% 21 Validation loss: 0.065790 Best loss: 0.047719 Accuracy: 98.63% 22 Validation loss: 0.045296 Best loss: 0.045296 Accuracy: 98.75% 23 Validation loss: 0.067558 Best loss: 0.045296 Accuracy: 98.48% 24 Validation loss: 0.052247 Best loss: 0.045296 Accuracy: 98.83% 25 Validation loss: 0.094011 Best loss: 0.045296 Accuracy: 98.51% 26 Validation loss: 0.064289 Best loss: 0.045296 Accuracy: 98.83% 27 Validation loss: 0.067531 Best loss: 0.045296 Accuracy: 98.63% 28 Validation loss: 0.066206 Best loss: 0.045296 Accuracy: 98.67% 29 Validation loss: 0.034466 Best loss: 0.034466 Accuracy: 99.10% 30 Validation loss: 0.049438 Best loss: 0.034466 Accuracy: 99.18% 31 Validation loss: 0.070736 Best loss: 0.034466 Accuracy: 98.71% 32 Validation loss: 0.045402 Best loss: 0.034466 Accuracy: 99.02% 33 Validation loss: 0.059611 Best loss: 0.034466 Accuracy: 99.06% 34 Validation loss: 0.053817 Best loss: 0.034466 Accuracy: 99.22% 35 Validation loss: 0.063382 Best loss: 0.034466 Accuracy: 99.18% 36 Validation loss: 0.068923 Best loss: 0.034466 Accuracy: 99.10% 37 Validation loss: 0.070492 Best loss: 0.034466 Accuracy: 98.98% 38 Validation loss: 0.074036 Best loss: 0.034466 Accuracy: 98.83% 39 Validation loss: 0.065798 Best loss: 0.034466 Accuracy: 99.06% 40 Validation loss: 0.076773 Best loss: 0.034466 Accuracy: 98.71% 41 Validation loss: 0.074852 Best loss: 0.034466 Accuracy: 98.79% 42 Validation loss: 0.064076 Best loss: 0.034466 Accuracy: 98.83% 43 Validation loss: 0.097832 Best loss: 0.034466 Accuracy: 98.59% 44 Validation loss: 0.071520 Best loss: 0.034466 Accuracy: 98.71% 45 Validation loss: 0.080763 Best loss: 0.034466 Accuracy: 98.83% 46 Validation loss: 0.089595 Best loss: 0.034466 Accuracy: 98.51% 47 Validation loss: 0.074774 Best loss: 0.034466 Accuracy: 98.91% 48 Validation loss: 0.119814 Best loss: 0.034466 Accuracy: 98.28% 49 Validation loss: 0.618709 Best loss: 0.034466 Accuracy: 95.00% 50 Validation loss: 1.004824 Best loss: 0.034466 Accuracy: 76.00% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 6.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1 0 Validation loss: 1.637596 Best loss: 1.637596 Accuracy: 19.27% 1 Validation loss: 1.621896 Best loss: 1.621896 Accuracy: 19.27% 2 Validation loss: 1.619832 Best loss: 1.619832 Accuracy: 19.27% 3 Validation loss: 1.616522 Best loss: 1.616522 Accuracy: 19.27% 4 Validation loss: 1.654961 Best loss: 1.616522 Accuracy: 22.01% 5 Validation loss: 1.621477 Best loss: 1.616522 Accuracy: 22.01% 6 Validation loss: 1.632694 Best loss: 1.616522 Accuracy: 19.27% 7 Validation loss: 1.614864 Best loss: 1.614864 Accuracy: 19.27% 8 Validation loss: 1.642610 Best loss: 1.614864 Accuracy: 19.27% 9 Validation loss: 1.639825 Best loss: 1.614864 Accuracy: 19.27% 10 Validation loss: 1.615379 Best loss: 1.614864 Accuracy: 22.01% 11 Validation loss: 1.630566 Best loss: 1.614864 Accuracy: 22.01% 12 Validation loss: 1.636741 Best loss: 1.614864 Accuracy: 19.08% 13 Validation loss: 1.613105 Best loss: 1.613105 Accuracy: 19.27% 14 Validation loss: 1.629265 Best loss: 1.613105 Accuracy: 19.08% 15 Validation loss: 1.617145 Best loss: 1.613105 Accuracy: 22.01% 16 Validation loss: 1.626926 Best loss: 1.613105 Accuracy: 18.73% 17 Validation loss: 1.608595 Best loss: 1.608595 Accuracy: 22.01% 18 Validation loss: 1.624733 Best loss: 1.608595 Accuracy: 19.27% 19 Validation loss: 1.624033 Best loss: 1.608595 Accuracy: 20.91% 20 Validation loss: 1.615446 Best loss: 1.608595 Accuracy: 19.27% 21 Validation loss: 1.629647 Best loss: 1.608595 Accuracy: 19.27% 22 Validation loss: 1.619092 Best loss: 1.608595 Accuracy: 19.27% 23 Validation loss: 1.621078 Best loss: 1.608595 Accuracy: 22.01% 24 Validation loss: 1.633986 Best loss: 1.608595 Accuracy: 22.01% 25 Validation loss: 1.647180 Best loss: 1.608595 Accuracy: 22.01% 26 Validation loss: 1.614127 Best loss: 1.608595 Accuracy: 19.08% 27 Validation loss: 1.638544 Best loss: 1.608595 Accuracy: 19.27% 28 Validation loss: 1.656266 Best loss: 1.608595 Accuracy: 22.01% 29 Validation loss: 1.617746 Best loss: 1.608595 Accuracy: 18.73% 30 Validation loss: 1.641872 Best loss: 1.608595 Accuracy: 19.27% 31 Validation loss: 1.631482 Best loss: 1.608595 Accuracy: 18.73% 32 Validation loss: 1.641180 Best loss: 1.608595 Accuracy: 18.73% 33 Validation loss: 1.647448 Best loss: 1.608595 Accuracy: 19.08% 34 Validation loss: 1.610285 Best loss: 1.608595 Accuracy: 22.01% 35 Validation loss: 1.624696 Best loss: 1.608595 Accuracy: 19.08% 36 Validation loss: 1.630856 Best loss: 1.608595 Accuracy: 18.73% 37 Validation loss: 1.632279 Best loss: 1.608595 Accuracy: 18.73% 38 Validation loss: 1.631023 Best loss: 1.608595 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1, total= 2.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1 0 Validation loss: 1.631925 Best loss: 1.631925 Accuracy: 22.01% 1 Validation loss: 1.644835 Best loss: 1.631925 Accuracy: 19.08% 2 Validation loss: 1.611659 Best loss: 1.611659 Accuracy: 22.01% 3 Validation loss: 1.614181 Best loss: 1.611659 Accuracy: 22.01% 4 Validation loss: 1.617812 Best loss: 1.611659 Accuracy: 22.01% 5 Validation loss: 1.624094 Best loss: 1.611659 Accuracy: 22.01% 6 Validation loss: 1.635103 Best loss: 1.611659 Accuracy: 19.27% 7 Validation loss: 1.630915 Best loss: 1.611659 Accuracy: 18.73% 8 Validation loss: 1.639864 Best loss: 1.611659 Accuracy: 22.01% 9 Validation loss: 1.611896 Best loss: 1.611659 Accuracy: 20.91% 10 Validation loss: 1.613292 Best loss: 1.611659 Accuracy: 22.01% 11 Validation loss: 1.614463 Best loss: 1.611659 Accuracy: 22.01% 12 Validation loss: 1.615520 Best loss: 1.611659 Accuracy: 20.91% 13 Validation loss: 1.609592 Best loss: 1.609592 Accuracy: 22.01% 14 Validation loss: 1.628132 Best loss: 1.609592 Accuracy: 22.01% 15 Validation loss: 1.613725 Best loss: 1.609592 Accuracy: 19.08% 16 Validation loss: 1.621646 Best loss: 1.609592 Accuracy: 22.01% 17 Validation loss: 1.612428 Best loss: 1.609592 Accuracy: 19.08% 18 Validation loss: 1.622169 Best loss: 1.609592 Accuracy: 22.01% 19 Validation loss: 1.611822 Best loss: 1.609592 Accuracy: 22.01% 20 Validation loss: 1.612159 Best loss: 1.609592 Accuracy: 20.91% 21 Validation loss: 1.657980 Best loss: 1.609592 Accuracy: 22.01% 22 Validation loss: 1.643385 Best loss: 1.609592 Accuracy: 19.27% 23 Validation loss: 1.647554 Best loss: 1.609592 Accuracy: 22.01% 24 Validation loss: 1.652203 Best loss: 1.609592 Accuracy: 18.73% 25 Validation loss: 1.610976 Best loss: 1.609592 Accuracy: 20.91% 26 Validation loss: 1.614099 Best loss: 1.609592 Accuracy: 20.91% 27 Validation loss: 1.614368 Best loss: 1.609592 Accuracy: 20.91% 28 Validation loss: 1.642985 Best loss: 1.609592 Accuracy: 19.08% 29 Validation loss: 1.614041 Best loss: 1.609592 Accuracy: 19.08% 30 Validation loss: 1.609285 Best loss: 1.609285 Accuracy: 22.01% 31 Validation loss: 1.634891 Best loss: 1.609285 Accuracy: 18.73% 32 Validation loss: 1.614004 Best loss: 1.609285 Accuracy: 19.08% 33 Validation loss: 1.619316 Best loss: 1.609285 Accuracy: 19.08% 34 Validation loss: 1.611677 Best loss: 1.609285 Accuracy: 22.01% 35 Validation loss: 1.638077 Best loss: 1.609285 Accuracy: 20.91% 36 Validation loss: 1.621401 Best loss: 1.609285 Accuracy: 22.01% 37 Validation loss: 1.617569 Best loss: 1.609285 Accuracy: 22.01% 38 Validation loss: 1.620771 Best loss: 1.609285 Accuracy: 20.91% 39 Validation loss: 1.627337 Best loss: 1.609285 Accuracy: 19.27% 40 Validation loss: 1.607884 Best loss: 1.607884 Accuracy: 22.01% 41 Validation loss: 1.634035 Best loss: 1.607884 Accuracy: 22.01% 42 Validation loss: 1.618333 Best loss: 1.607884 Accuracy: 20.91% 43 Validation loss: 1.618746 Best loss: 1.607884 Accuracy: 22.01% 44 Validation loss: 1.621663 Best loss: 1.607884 Accuracy: 19.08% 45 Validation loss: 1.629384 Best loss: 1.607884 Accuracy: 22.01% 46 Validation loss: 1.637659 Best loss: 1.607884 Accuracy: 18.73% 47 Validation loss: 1.609606 Best loss: 1.607884 Accuracy: 22.01% 48 Validation loss: 1.612162 Best loss: 1.607884 Accuracy: 22.01% 49 Validation loss: 1.643129 Best loss: 1.607884 Accuracy: 19.27% 50 Validation loss: 1.611989 Best loss: 1.607884 Accuracy: 22.01% 51 Validation loss: 1.619973 Best loss: 1.607884 Accuracy: 19.08% 52 Validation loss: 1.627020 Best loss: 1.607884 Accuracy: 19.08% 53 Validation loss: 1.618522 Best loss: 1.607884 Accuracy: 22.01% 54 Validation loss: 1.654810 Best loss: 1.607884 Accuracy: 18.73% 55 Validation loss: 1.645115 Best loss: 1.607884 Accuracy: 18.73% 56 Validation loss: 1.669472 Best loss: 1.607884 Accuracy: 19.08% 57 Validation loss: 1.619669 Best loss: 1.607884 Accuracy: 18.73% 58 Validation loss: 1.612547 Best loss: 1.607884 Accuracy: 22.01% 59 Validation loss: 1.619514 Best loss: 1.607884 Accuracy: 22.01% 60 Validation loss: 1.635965 Best loss: 1.607884 Accuracy: 18.73% 61 Validation loss: 1.627400 Best loss: 1.607884 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1, total= 3.4min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1 0 Validation loss: 1.644674 Best loss: 1.644674 Accuracy: 19.27% 1 Validation loss: 1.624597 Best loss: 1.624597 Accuracy: 19.27% 2 Validation loss: 1.614160 Best loss: 1.614160 Accuracy: 19.27% 3 Validation loss: 1.616997 Best loss: 1.614160 Accuracy: 19.27% 4 Validation loss: 1.614504 Best loss: 1.614160 Accuracy: 22.01% 5 Validation loss: 1.627702 Best loss: 1.614160 Accuracy: 22.01% 6 Validation loss: 1.648679 Best loss: 1.614160 Accuracy: 19.27% 7 Validation loss: 1.627600 Best loss: 1.614160 Accuracy: 18.73% 8 Validation loss: 1.625684 Best loss: 1.614160 Accuracy: 22.01% 9 Validation loss: 1.615556 Best loss: 1.614160 Accuracy: 20.91% 10 Validation loss: 1.619122 Best loss: 1.614160 Accuracy: 22.01% 11 Validation loss: 1.639598 Best loss: 1.614160 Accuracy: 22.01% 12 Validation loss: 1.627612 Best loss: 1.614160 Accuracy: 19.08% 13 Validation loss: 1.615013 Best loss: 1.614160 Accuracy: 19.08% 14 Validation loss: 1.624332 Best loss: 1.614160 Accuracy: 19.08% 15 Validation loss: 1.612442 Best loss: 1.612442 Accuracy: 22.01% 16 Validation loss: 1.627837 Best loss: 1.612442 Accuracy: 22.01% 17 Validation loss: 1.613230 Best loss: 1.612442 Accuracy: 18.73% 18 Validation loss: 1.615751 Best loss: 1.612442 Accuracy: 22.01% 19 Validation loss: 1.619094 Best loss: 1.612442 Accuracy: 22.01% 20 Validation loss: 1.609109 Best loss: 1.609109 Accuracy: 22.01% 21 Validation loss: 1.689012 Best loss: 1.609109 Accuracy: 22.01% 22 Validation loss: 1.618890 Best loss: 1.609109 Accuracy: 18.73% 23 Validation loss: 1.638926 Best loss: 1.609109 Accuracy: 22.01% 24 Validation loss: 1.660615 Best loss: 1.609109 Accuracy: 20.91% 25 Validation loss: 1.619330 Best loss: 1.609109 Accuracy: 19.27% 26 Validation loss: 1.611331 Best loss: 1.609109 Accuracy: 22.01% 27 Validation loss: 1.614303 Best loss: 1.609109 Accuracy: 20.91% 28 Validation loss: 1.628117 Best loss: 1.609109 Accuracy: 19.08% 29 Validation loss: 1.610351 Best loss: 1.609109 Accuracy: 20.91% 30 Validation loss: 1.626937 Best loss: 1.609109 Accuracy: 20.91% 31 Validation loss: 1.633268 Best loss: 1.609109 Accuracy: 18.73% 32 Validation loss: 1.609064 Best loss: 1.609064 Accuracy: 22.01% 33 Validation loss: 1.610650 Best loss: 1.609064 Accuracy: 19.08% 34 Validation loss: 1.616982 Best loss: 1.609064 Accuracy: 22.01% 35 Validation loss: 1.627556 Best loss: 1.609064 Accuracy: 19.27% 36 Validation loss: 1.609165 Best loss: 1.609064 Accuracy: 20.91% 37 Validation loss: 1.626448 Best loss: 1.609064 Accuracy: 19.27% 38 Validation loss: 1.614501 Best loss: 1.609064 Accuracy: 22.01% 39 Validation loss: 1.617749 Best loss: 1.609064 Accuracy: 19.27% 40 Validation loss: 1.623126 Best loss: 1.609064 Accuracy: 22.01% 41 Validation loss: 1.615781 Best loss: 1.609064 Accuracy: 19.27% 42 Validation loss: 1.617480 Best loss: 1.609064 Accuracy: 19.27% 43 Validation loss: 1.617777 Best loss: 1.609064 Accuracy: 19.08% 44 Validation loss: 1.626857 Best loss: 1.609064 Accuracy: 22.01% 45 Validation loss: 1.613780 Best loss: 1.609064 Accuracy: 22.01% 46 Validation loss: 1.618689 Best loss: 1.609064 Accuracy: 22.01% 47 Validation loss: 1.610631 Best loss: 1.609064 Accuracy: 22.01% 48 Validation loss: 1.614393 Best loss: 1.609064 Accuracy: 18.73% 49 Validation loss: 1.612179 Best loss: 1.609064 Accuracy: 19.27% 50 Validation loss: 1.623882 Best loss: 1.609064 Accuracy: 19.27% 51 Validation loss: 1.616773 Best loss: 1.609064 Accuracy: 22.01% 52 Validation loss: 1.620595 Best loss: 1.609064 Accuracy: 18.73% 53 Validation loss: 1.617682 Best loss: 1.609064 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=10, learning_rate=0.1, total= 2.9min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.105630 Best loss: 0.105630 Accuracy: 97.22% 1 Validation loss: 0.112534 Best loss: 0.105630 Accuracy: 97.26% 2 Validation loss: 0.094753 Best loss: 0.094753 Accuracy: 97.65% 3 Validation loss: 0.090513 Best loss: 0.090513 Accuracy: 97.22% 4 Validation loss: 0.090514 Best loss: 0.090513 Accuracy: 97.77% 5 Validation loss: 0.102136 Best loss: 0.090513 Accuracy: 97.65% 6 Validation loss: 0.171315 Best loss: 0.090513 Accuracy: 96.36% 7 Validation loss: 0.124246 Best loss: 0.090513 Accuracy: 97.65% 8 Validation loss: 0.094865 Best loss: 0.090513 Accuracy: 97.54% 9 Validation loss: 0.087033 Best loss: 0.087033 Accuracy: 97.93% 10 Validation loss: 0.105954 Best loss: 0.087033 Accuracy: 96.76% 11 Validation loss: 0.099992 Best loss: 0.087033 Accuracy: 97.42% 12 Validation loss: 0.117393 Best loss: 0.087033 Accuracy: 97.89% 13 Validation loss: 0.076170 Best loss: 0.076170 Accuracy: 98.24% 14 Validation loss: 0.105598 Best loss: 0.076170 Accuracy: 98.16% 15 Validation loss: 0.105871 Best loss: 0.076170 Accuracy: 98.01% 16 Validation loss: 0.096650 Best loss: 0.076170 Accuracy: 97.93% 17 Validation loss: 0.112779 Best loss: 0.076170 Accuracy: 97.26% 18 Validation loss: 0.140539 Best loss: 0.076170 Accuracy: 97.38% 19 Validation loss: 0.112113 Best loss: 0.076170 Accuracy: 98.05% 20 Validation loss: 0.105863 Best loss: 0.076170 Accuracy: 98.01% 21 Validation loss: 0.132844 Best loss: 0.076170 Accuracy: 96.91% 22 Validation loss: 0.098543 Best loss: 0.076170 Accuracy: 97.65% 23 Validation loss: 0.137065 Best loss: 0.076170 Accuracy: 97.22% 24 Validation loss: 0.117392 Best loss: 0.076170 Accuracy: 97.77% 25 Validation loss: 0.086141 Best loss: 0.076170 Accuracy: 98.36% 26 Validation loss: 0.097466 Best loss: 0.076170 Accuracy: 97.97% 27 Validation loss: 0.100154 Best loss: 0.076170 Accuracy: 97.85% 28 Validation loss: 0.098387 Best loss: 0.076170 Accuracy: 98.16% 29 Validation loss: 0.098656 Best loss: 0.076170 Accuracy: 98.12% 30 Validation loss: 0.111373 Best loss: 0.076170 Accuracy: 97.77% 31 Validation loss: 0.127052 Best loss: 0.076170 Accuracy: 97.85% 32 Validation loss: 0.109520 Best loss: 0.076170 Accuracy: 98.12% 33 Validation loss: 0.101836 Best loss: 0.076170 Accuracy: 98.24% 34 Validation loss: 0.129367 Best loss: 0.076170 Accuracy: 98.20% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 13.8s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.111783 Best loss: 0.111783 Accuracy: 97.03% 1 Validation loss: 0.089338 Best loss: 0.089338 Accuracy: 97.62% 2 Validation loss: 0.104556 Best loss: 0.089338 Accuracy: 96.91% 3 Validation loss: 0.083850 Best loss: 0.083850 Accuracy: 97.69% 4 Validation loss: 0.093423 Best loss: 0.083850 Accuracy: 97.73% 5 Validation loss: 0.090041 Best loss: 0.083850 Accuracy: 97.22% 6 Validation loss: 0.088448 Best loss: 0.083850 Accuracy: 97.69% 7 Validation loss: 0.086538 Best loss: 0.083850 Accuracy: 97.46% 8 Validation loss: 0.104112 Best loss: 0.083850 Accuracy: 97.77% 9 Validation loss: 0.094806 Best loss: 0.083850 Accuracy: 97.69% 10 Validation loss: 0.087629 Best loss: 0.083850 Accuracy: 97.81% 11 Validation loss: 0.082494 Best loss: 0.082494 Accuracy: 97.93% 12 Validation loss: 0.089536 Best loss: 0.082494 Accuracy: 97.93% 13 Validation loss: 0.100433 Best loss: 0.082494 Accuracy: 98.05% 14 Validation loss: 0.089536 Best loss: 0.082494 Accuracy: 98.36% 15 Validation loss: 0.119185 Best loss: 0.082494 Accuracy: 98.08% 16 Validation loss: 0.118379 Best loss: 0.082494 Accuracy: 97.85% 17 Validation loss: 0.095182 Best loss: 0.082494 Accuracy: 97.93% 18 Validation loss: 0.083229 Best loss: 0.082494 Accuracy: 97.77% 19 Validation loss: 0.144655 Best loss: 0.082494 Accuracy: 96.05% 20 Validation loss: 0.080189 Best loss: 0.080189 Accuracy: 97.65% 21 Validation loss: 0.091286 Best loss: 0.080189 Accuracy: 98.01% 22 Validation loss: 0.078845 Best loss: 0.078845 Accuracy: 98.05% 23 Validation loss: 0.097417 Best loss: 0.078845 Accuracy: 98.16% 24 Validation loss: 0.091532 Best loss: 0.078845 Accuracy: 98.36% 25 Validation loss: 0.089397 Best loss: 0.078845 Accuracy: 98.12% 26 Validation loss: 0.092321 Best loss: 0.078845 Accuracy: 98.08% 27 Validation loss: 0.098038 Best loss: 0.078845 Accuracy: 97.97% 28 Validation loss: 0.112625 Best loss: 0.078845 Accuracy: 98.01% 29 Validation loss: 0.111579 Best loss: 0.078845 Accuracy: 98.01% 30 Validation loss: 0.141701 Best loss: 0.078845 Accuracy: 97.89% 31 Validation loss: 0.116426 Best loss: 0.078845 Accuracy: 97.73% 32 Validation loss: 0.099619 Best loss: 0.078845 Accuracy: 97.97% 33 Validation loss: 0.109189 Best loss: 0.078845 Accuracy: 97.77% 34 Validation loss: 0.111339 Best loss: 0.078845 Accuracy: 97.62% 35 Validation loss: 0.085465 Best loss: 0.078845 Accuracy: 98.16% 36 Validation loss: 0.112406 Best loss: 0.078845 Accuracy: 97.93% 37 Validation loss: 0.105894 Best loss: 0.078845 Accuracy: 98.08% 38 Validation loss: 0.092110 Best loss: 0.078845 Accuracy: 98.05% 39 Validation loss: 0.107523 Best loss: 0.078845 Accuracy: 97.46% 40 Validation loss: 0.074957 Best loss: 0.074957 Accuracy: 98.20% 41 Validation loss: 0.175366 Best loss: 0.074957 Accuracy: 97.22% 42 Validation loss: 0.161186 Best loss: 0.074957 Accuracy: 96.60% 43 Validation loss: 0.150176 Best loss: 0.074957 Accuracy: 96.95% 44 Validation loss: 0.156384 Best loss: 0.074957 Accuracy: 97.07% 45 Validation loss: 0.119757 Best loss: 0.074957 Accuracy: 97.73% 46 Validation loss: 0.174601 Best loss: 0.074957 Accuracy: 97.07% 47 Validation loss: 0.134940 Best loss: 0.074957 Accuracy: 97.97% 48 Validation loss: 0.109069 Best loss: 0.074957 Accuracy: 98.24% 49 Validation loss: 0.122947 Best loss: 0.074957 Accuracy: 98.24% 50 Validation loss: 0.108685 Best loss: 0.074957 Accuracy: 98.24% 51 Validation loss: 0.110896 Best loss: 0.074957 Accuracy: 98.12% 52 Validation loss: 0.110465 Best loss: 0.074957 Accuracy: 98.12% 53 Validation loss: 0.154313 Best loss: 0.074957 Accuracy: 97.50% 54 Validation loss: 0.161417 Best loss: 0.074957 Accuracy: 97.65% 55 Validation loss: 0.174896 Best loss: 0.074957 Accuracy: 97.73% 56 Validation loss: 0.118686 Best loss: 0.074957 Accuracy: 97.81% 57 Validation loss: 0.108798 Best loss: 0.074957 Accuracy: 97.77% 58 Validation loss: 0.144606 Best loss: 0.074957 Accuracy: 97.34% 59 Validation loss: 0.098687 Best loss: 0.074957 Accuracy: 98.08% 60 Validation loss: 0.105361 Best loss: 0.074957 Accuracy: 98.28% 61 Validation loss: 0.099564 Best loss: 0.074957 Accuracy: 98.16% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 24.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.092839 Best loss: 0.092839 Accuracy: 97.46% 1 Validation loss: 0.096513 Best loss: 0.092839 Accuracy: 97.34% 2 Validation loss: 0.087615 Best loss: 0.087615 Accuracy: 97.65% 3 Validation loss: 0.063588 Best loss: 0.063588 Accuracy: 98.40% 4 Validation loss: 0.076326 Best loss: 0.063588 Accuracy: 97.77% 5 Validation loss: 0.077270 Best loss: 0.063588 Accuracy: 97.73% 6 Validation loss: 0.072396 Best loss: 0.063588 Accuracy: 98.16% 7 Validation loss: 0.104660 Best loss: 0.063588 Accuracy: 97.15% 8 Validation loss: 0.108871 Best loss: 0.063588 Accuracy: 97.65% 9 Validation loss: 0.083605 Best loss: 0.063588 Accuracy: 98.01% 10 Validation loss: 0.075325 Best loss: 0.063588 Accuracy: 98.12% 11 Validation loss: 0.083171 Best loss: 0.063588 Accuracy: 98.48% 12 Validation loss: 0.080781 Best loss: 0.063588 Accuracy: 98.16% 13 Validation loss: 0.096771 Best loss: 0.063588 Accuracy: 98.20% 14 Validation loss: 0.081002 Best loss: 0.063588 Accuracy: 98.20% 15 Validation loss: 0.093805 Best loss: 0.063588 Accuracy: 98.24% 16 Validation loss: 0.080396 Best loss: 0.063588 Accuracy: 98.44% 17 Validation loss: 0.173369 Best loss: 0.063588 Accuracy: 97.11% 18 Validation loss: 0.091176 Best loss: 0.063588 Accuracy: 98.24% 19 Validation loss: 0.081581 Best loss: 0.063588 Accuracy: 98.40% 20 Validation loss: 0.120644 Best loss: 0.063588 Accuracy: 97.73% 21 Validation loss: 0.084396 Best loss: 0.063588 Accuracy: 98.55% 22 Validation loss: 0.093213 Best loss: 0.063588 Accuracy: 98.24% 23 Validation loss: 0.090317 Best loss: 0.063588 Accuracy: 98.12% 24 Validation loss: 0.104277 Best loss: 0.063588 Accuracy: 97.81% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 10.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 1.863424 Best loss: 1.863424 Accuracy: 62.51% 1 Validation loss: 0.178661 Best loss: 0.178661 Accuracy: 94.53% 2 Validation loss: 0.140918 Best loss: 0.140918 Accuracy: 95.62% 3 Validation loss: 0.106734 Best loss: 0.106734 Accuracy: 96.40% 4 Validation loss: 604.304810 Best loss: 0.106734 Accuracy: 18.73% 5 Validation loss: 2.040188 Best loss: 0.106734 Accuracy: 50.70% 6 Validation loss: 1.729430 Best loss: 0.106734 Accuracy: 50.51% 7 Validation loss: 1.155580 Best loss: 0.106734 Accuracy: 62.71% 8 Validation loss: 1.046638 Best loss: 0.106734 Accuracy: 66.46% 9 Validation loss: 1.327363 Best loss: 0.106734 Accuracy: 60.71% 10 Validation loss: 0.889922 Best loss: 0.106734 Accuracy: 71.77% 11 Validation loss: 0.900192 Best loss: 0.106734 Accuracy: 70.76% 12 Validation loss: 0.982543 Best loss: 0.106734 Accuracy: 68.26% 13 Validation loss: 0.942618 Best loss: 0.106734 Accuracy: 68.69% 14 Validation loss: 0.871513 Best loss: 0.106734 Accuracy: 71.34% 15 Validation loss: 0.688626 Best loss: 0.106734 Accuracy: 76.70% 16 Validation loss: 0.591786 Best loss: 0.106734 Accuracy: 79.40% 17 Validation loss: 0.591854 Best loss: 0.106734 Accuracy: 80.34% 18 Validation loss: 0.629652 Best loss: 0.106734 Accuracy: 79.71% 19 Validation loss: 0.654583 Best loss: 0.106734 Accuracy: 80.02% 20 Validation loss: 0.508119 Best loss: 0.106734 Accuracy: 83.27% 21 Validation loss: 0.461316 Best loss: 0.106734 Accuracy: 84.13% 22 Validation loss: 0.432319 Best loss: 0.106734 Accuracy: 85.93% 23 Validation loss: 1.691923 Best loss: 0.106734 Accuracy: 86.36% 24 Validation loss: 0.384710 Best loss: 0.106734 Accuracy: 87.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 11.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 0.205389 Best loss: 0.205389 Accuracy: 94.18% 1 Validation loss: 1.403126 Best loss: 0.205389 Accuracy: 78.34% 2 Validation loss: 5.576907 Best loss: 0.205389 Accuracy: 64.46% 3 Validation loss: 1.470547 Best loss: 0.205389 Accuracy: 65.48% 4 Validation loss: 1.085397 Best loss: 0.205389 Accuracy: 67.32% 5 Validation loss: 0.581014 Best loss: 0.205389 Accuracy: 82.60% 6 Validation loss: 0.490068 Best loss: 0.205389 Accuracy: 85.22% 7 Validation loss: 0.424210 Best loss: 0.205389 Accuracy: 87.76% 8 Validation loss: 0.377084 Best loss: 0.205389 Accuracy: 88.15% 9 Validation loss: 0.338571 Best loss: 0.205389 Accuracy: 90.58% 10 Validation loss: 0.281597 Best loss: 0.205389 Accuracy: 91.63% 11 Validation loss: 0.233957 Best loss: 0.205389 Accuracy: 93.75% 12 Validation loss: 0.215218 Best loss: 0.205389 Accuracy: 94.06% 13 Validation loss: 0.225218 Best loss: 0.205389 Accuracy: 94.37% 14 Validation loss: 0.224049 Best loss: 0.205389 Accuracy: 94.49% 15 Validation loss: 0.217113 Best loss: 0.205389 Accuracy: 94.06% 16 Validation loss: 0.194696 Best loss: 0.194696 Accuracy: 94.72% 17 Validation loss: 0.229181 Best loss: 0.194696 Accuracy: 93.43% 18 Validation loss: 0.179172 Best loss: 0.179172 Accuracy: 95.62% 19 Validation loss: 0.364202 Best loss: 0.179172 Accuracy: 91.36% 20 Validation loss: 46.370445 Best loss: 0.179172 Accuracy: 46.91% 21 Validation loss: 3.306267 Best loss: 0.179172 Accuracy: 62.43% 22 Validation loss: 2.370848 Best loss: 0.179172 Accuracy: 66.65% 23 Validation loss: 2.553179 Best loss: 0.179172 Accuracy: 65.83% 24 Validation loss: 3.606383 Best loss: 0.179172 Accuracy: 60.24% 25 Validation loss: 1.053476 Best loss: 0.179172 Accuracy: 69.94% 26 Validation loss: 3.161819 Best loss: 0.179172 Accuracy: 68.96% 27 Validation loss: 1.566990 Best loss: 0.179172 Accuracy: 66.38% 28 Validation loss: 38.209747 Best loss: 0.179172 Accuracy: 30.53% 29 Validation loss: 3.868488 Best loss: 0.179172 Accuracy: 53.79% 30 Validation loss: 1.781909 Best loss: 0.179172 Accuracy: 68.73% 31 Validation loss: 1.606874 Best loss: 0.179172 Accuracy: 68.96% 32 Validation loss: 0.870383 Best loss: 0.179172 Accuracy: 75.25% 33 Validation loss: 0.857675 Best loss: 0.179172 Accuracy: 75.18% 34 Validation loss: 0.770050 Best loss: 0.179172 Accuracy: 76.15% 35 Validation loss: 0.847934 Best loss: 0.179172 Accuracy: 76.39% 36 Validation loss: 0.760057 Best loss: 0.179172 Accuracy: 74.94% 37 Validation loss: 0.800689 Best loss: 0.179172 Accuracy: 77.44% 38 Validation loss: 0.521493 Best loss: 0.179172 Accuracy: 83.07% 39 Validation loss: 0.775157 Best loss: 0.179172 Accuracy: 78.62% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 16.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 0.193115 Best loss: 0.193115 Accuracy: 95.47% 1 Validation loss: 3.285612 Best loss: 0.193115 Accuracy: 63.96% 2 Validation loss: 4.193580 Best loss: 0.193115 Accuracy: 56.61% 3 Validation loss: 1.217634 Best loss: 0.193115 Accuracy: 57.39% 4 Validation loss: 0.783022 Best loss: 0.193115 Accuracy: 70.80% 5 Validation loss: 0.875647 Best loss: 0.193115 Accuracy: 68.18% 6 Validation loss: 0.900572 Best loss: 0.193115 Accuracy: 63.92% 7 Validation loss: 0.700143 Best loss: 0.193115 Accuracy: 76.15% 8 Validation loss: 0.578958 Best loss: 0.193115 Accuracy: 78.30% 9 Validation loss: 0.526057 Best loss: 0.193115 Accuracy: 80.92% 10 Validation loss: 0.646923 Best loss: 0.193115 Accuracy: 77.56% 11 Validation loss: 0.563426 Best loss: 0.193115 Accuracy: 80.41% 12 Validation loss: 0.408865 Best loss: 0.193115 Accuracy: 85.97% 13 Validation loss: 0.487587 Best loss: 0.193115 Accuracy: 82.56% 14 Validation loss: 0.363604 Best loss: 0.193115 Accuracy: 88.51% 15 Validation loss: 0.357173 Best loss: 0.193115 Accuracy: 88.15% 16 Validation loss: 0.356857 Best loss: 0.193115 Accuracy: 88.55% 17 Validation loss: 0.330360 Best loss: 0.193115 Accuracy: 89.33% 18 Validation loss: 0.271285 Best loss: 0.193115 Accuracy: 91.40% 19 Validation loss: 0.236158 Best loss: 0.193115 Accuracy: 91.79% 20 Validation loss: 0.230091 Best loss: 0.193115 Accuracy: 92.53% 21 Validation loss: 0.218871 Best loss: 0.193115 Accuracy: 92.65% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 10.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01 0 Validation loss: 0.088623 Best loss: 0.088623 Accuracy: 97.50% 1 Validation loss: 0.091316 Best loss: 0.088623 Accuracy: 97.77% 2 Validation loss: 0.083109 Best loss: 0.083109 Accuracy: 97.89% 3 Validation loss: 0.084447 Best loss: 0.083109 Accuracy: 97.54% 4 Validation loss: 0.073347 Best loss: 0.073347 Accuracy: 98.12% 5 Validation loss: 0.067175 Best loss: 0.067175 Accuracy: 98.20% 6 Validation loss: 0.150697 Best loss: 0.067175 Accuracy: 96.40% 7 Validation loss: 0.099371 Best loss: 0.067175 Accuracy: 97.50% 8 Validation loss: 0.091203 Best loss: 0.067175 Accuracy: 98.01% 9 Validation loss: 0.076368 Best loss: 0.067175 Accuracy: 98.32% 10 Validation loss: 0.123249 Best loss: 0.067175 Accuracy: 97.69% 11 Validation loss: 0.071085 Best loss: 0.067175 Accuracy: 98.59% 12 Validation loss: 0.097651 Best loss: 0.067175 Accuracy: 97.69% 13 Validation loss: 0.242061 Best loss: 0.067175 Accuracy: 95.78% 14 Validation loss: 0.124799 Best loss: 0.067175 Accuracy: 97.93% 15 Validation loss: 0.098963 Best loss: 0.067175 Accuracy: 98.05% 16 Validation loss: 0.106303 Best loss: 0.067175 Accuracy: 98.44% 17 Validation loss: 0.083130 Best loss: 0.067175 Accuracy: 98.55% 18 Validation loss: 0.236050 Best loss: 0.067175 Accuracy: 98.08% 19 Validation loss: 0.135998 Best loss: 0.067175 Accuracy: 98.12% 20 Validation loss: 0.291852 Best loss: 0.067175 Accuracy: 98.05% 21 Validation loss: 1.931169 Best loss: 0.067175 Accuracy: 97.42% 22 Validation loss: 0.270029 Best loss: 0.067175 Accuracy: 97.69% 23 Validation loss: 0.243325 Best loss: 0.067175 Accuracy: 98.44% 24 Validation loss: 0.225155 Best loss: 0.067175 Accuracy: 98.05% 25 Validation loss: 0.243621 Best loss: 0.067175 Accuracy: 98.20% 26 Validation loss: 0.248698 Best loss: 0.067175 Accuracy: 98.05% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01, total= 25.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01 0 Validation loss: 0.086239 Best loss: 0.086239 Accuracy: 97.54% 1 Validation loss: 0.101958 Best loss: 0.086239 Accuracy: 98.01% 2 Validation loss: 0.067972 Best loss: 0.067972 Accuracy: 98.32% 3 Validation loss: 0.132947 Best loss: 0.067972 Accuracy: 97.50% 4 Validation loss: 0.079015 Best loss: 0.067972 Accuracy: 97.97% 5 Validation loss: 0.106633 Best loss: 0.067972 Accuracy: 97.85% 6 Validation loss: 0.074450 Best loss: 0.067972 Accuracy: 98.51% 7 Validation loss: 1.089252 Best loss: 0.067972 Accuracy: 96.09% 8 Validation loss: 0.187703 Best loss: 0.067972 Accuracy: 97.81% 9 Validation loss: 1.214255 Best loss: 0.067972 Accuracy: 94.37% 10 Validation loss: 0.154307 Best loss: 0.067972 Accuracy: 98.20% 11 Validation loss: 0.116002 Best loss: 0.067972 Accuracy: 98.48% 12 Validation loss: 0.102167 Best loss: 0.067972 Accuracy: 98.28% 13 Validation loss: 0.189098 Best loss: 0.067972 Accuracy: 98.20% 14 Validation loss: 0.103790 Best loss: 0.067972 Accuracy: 98.28% 15 Validation loss: 0.121718 Best loss: 0.067972 Accuracy: 98.59% 16 Validation loss: 0.119028 Best loss: 0.067972 Accuracy: 98.32% 17 Validation loss: 0.186052 Best loss: 0.067972 Accuracy: 98.44% 18 Validation loss: 0.441885 Best loss: 0.067972 Accuracy: 96.01% 19 Validation loss: 0.131240 Best loss: 0.067972 Accuracy: 97.65% 20 Validation loss: 0.322820 Best loss: 0.067972 Accuracy: 96.79% 21 Validation loss: 0.176219 Best loss: 0.067972 Accuracy: 98.12% 22 Validation loss: 0.164293 Best loss: 0.067972 Accuracy: 97.73% 23 Validation loss: 0.276100 Best loss: 0.067972 Accuracy: 98.28% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01, total= 21.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01 0 Validation loss: 0.085082 Best loss: 0.085082 Accuracy: 97.85% 1 Validation loss: 0.082635 Best loss: 0.082635 Accuracy: 97.93% 2 Validation loss: 0.087608 Best loss: 0.082635 Accuracy: 97.85% 3 Validation loss: 0.119522 Best loss: 0.082635 Accuracy: 96.99% 4 Validation loss: 0.060845 Best loss: 0.060845 Accuracy: 98.20% 5 Validation loss: 0.077505 Best loss: 0.060845 Accuracy: 98.16% 6 Validation loss: 0.080038 Best loss: 0.060845 Accuracy: 98.36% 7 Validation loss: 0.095316 Best loss: 0.060845 Accuracy: 98.20% 8 Validation loss: 0.072334 Best loss: 0.060845 Accuracy: 98.48% 9 Validation loss: 0.081715 Best loss: 0.060845 Accuracy: 98.05% 10 Validation loss: 0.078974 Best loss: 0.060845 Accuracy: 98.24% 11 Validation loss: 0.133812 Best loss: 0.060845 Accuracy: 98.16% 12 Validation loss: 0.607058 Best loss: 0.060845 Accuracy: 97.15% 13 Validation loss: 0.114474 Best loss: 0.060845 Accuracy: 97.65% 14 Validation loss: 0.096175 Best loss: 0.060845 Accuracy: 98.12% 15 Validation loss: 0.075544 Best loss: 0.060845 Accuracy: 98.24% 16 Validation loss: 0.077252 Best loss: 0.060845 Accuracy: 98.32% 17 Validation loss: 0.086365 Best loss: 0.060845 Accuracy: 98.05% 18 Validation loss: 0.077185 Best loss: 0.060845 Accuracy: 98.59% 19 Validation loss: 0.080713 Best loss: 0.060845 Accuracy: 98.48% 20 Validation loss: 7.787071 Best loss: 0.060845 Accuracy: 87.84% 21 Validation loss: 0.761765 Best loss: 0.060845 Accuracy: 98.01% 22 Validation loss: 0.787678 Best loss: 0.060845 Accuracy: 97.38% 23 Validation loss: 0.816786 Best loss: 0.060845 Accuracy: 98.05% 24 Validation loss: 0.925729 Best loss: 0.060845 Accuracy: 97.97% 25 Validation loss: 1.011159 Best loss: 0.060845 Accuracy: 98.08% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=50, learning_rate=0.01, total= 22.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02 0 Validation loss: 8.398409 Best loss: 8.398409 Accuracy: 89.37% 1 Validation loss: 0.911810 Best loss: 0.911810 Accuracy: 94.29% 2 Validation loss: 1373.153809 Best loss: 0.911810 Accuracy: 90.77% 3 Validation loss: 292.131073 Best loss: 0.911810 Accuracy: 90.81% 4 Validation loss: 1569.323486 Best loss: 0.911810 Accuracy: 95.58% 5 Validation loss: 540.995483 Best loss: 0.911810 Accuracy: 95.62% 6 Validation loss: 229.726318 Best loss: 0.911810 Accuracy: 96.91% 7 Validation loss: 23803.324219 Best loss: 0.911810 Accuracy: 84.79% 8 Validation loss: 1208.687500 Best loss: 0.911810 Accuracy: 95.50% 9 Validation loss: 279.935974 Best loss: 0.911810 Accuracy: 96.21% 10 Validation loss: 11762.333008 Best loss: 0.911810 Accuracy: 96.01% 11 Validation loss: 2459.948975 Best loss: 0.911810 Accuracy: 96.64% 12 Validation loss: 1706.548828 Best loss: 0.911810 Accuracy: 90.27% 13 Validation loss: 1394.444580 Best loss: 0.911810 Accuracy: 94.84% 14 Validation loss: 904.659485 Best loss: 0.911810 Accuracy: 97.30% 15 Validation loss: 788.806213 Best loss: 0.911810 Accuracy: 97.38% 16 Validation loss: 2277.578613 Best loss: 0.911810 Accuracy: 96.95% 17 Validation loss: 3479.282959 Best loss: 0.911810 Accuracy: 95.93% 18 Validation loss: 2845.820801 Best loss: 0.911810 Accuracy: 97.30% 19 Validation loss: 2507.192383 Best loss: 0.911810 Accuracy: 97.65% 20 Validation loss: 5944.325195 Best loss: 0.911810 Accuracy: 96.01% 21 Validation loss: 2033.826416 Best loss: 0.911810 Accuracy: 97.07% 22 Validation loss: 2066.968018 Best loss: 0.911810 Accuracy: 97.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02, total= 1.7min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02 0 Validation loss: 16.002684 Best loss: 16.002684 Accuracy: 77.72% 1 Validation loss: 1726.773438 Best loss: 16.002684 Accuracy: 80.02% 2 Validation loss: 650.942383 Best loss: 16.002684 Accuracy: 91.48% 3 Validation loss: 261.625000 Best loss: 16.002684 Accuracy: 95.35% 4 Validation loss: 8581.472656 Best loss: 16.002684 Accuracy: 72.52% 5 Validation loss: 2197.062988 Best loss: 16.002684 Accuracy: 95.74% 6 Validation loss: 713.716492 Best loss: 16.002684 Accuracy: 96.79% 7 Validation loss: 646.197876 Best loss: 16.002684 Accuracy: 94.92% 8 Validation loss: 263.925629 Best loss: 16.002684 Accuracy: 97.11% 9 Validation loss: 3297.005859 Best loss: 16.002684 Accuracy: 89.48% 10 Validation loss: 1239.920044 Best loss: 16.002684 Accuracy: 95.35% 11 Validation loss: 2979.242188 Best loss: 16.002684 Accuracy: 95.97% 12 Validation loss: 901.591064 Best loss: 16.002684 Accuracy: 97.30% 13 Validation loss: 610.036682 Best loss: 16.002684 Accuracy: 95.82% 14 Validation loss: 3353.314697 Best loss: 16.002684 Accuracy: 97.65% 15 Validation loss: 7475.214844 Best loss: 16.002684 Accuracy: 96.01% 16 Validation loss: 5848.258301 Best loss: 16.002684 Accuracy: 97.38% 17 Validation loss: 11151.689453 Best loss: 16.002684 Accuracy: 97.54% 18 Validation loss: 5717.995117 Best loss: 16.002684 Accuracy: 96.76% 19 Validation loss: 4425.187012 Best loss: 16.002684 Accuracy: 97.34% 20 Validation loss: 24611.513672 Best loss: 16.002684 Accuracy: 89.91% 21 Validation loss: 2946.307617 Best loss: 16.002684 Accuracy: 97.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02, total= 1.5min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02 0 Validation loss: 7844.526367 Best loss: 7844.526367 Accuracy: 72.60% 1 Validation loss: 102.427071 Best loss: 102.427071 Accuracy: 92.53% 2 Validation loss: 347.699677 Best loss: 102.427071 Accuracy: 90.15% 3 Validation loss: 55.027634 Best loss: 55.027634 Accuracy: 95.62% 4 Validation loss: 9237.209961 Best loss: 55.027634 Accuracy: 73.26% 5 Validation loss: 89.678726 Best loss: 55.027634 Accuracy: 92.53% 6 Validation loss: 13.722250 Best loss: 13.722250 Accuracy: 96.72% 7 Validation loss: 448.204102 Best loss: 13.722250 Accuracy: 96.52% 8 Validation loss: 634.872314 Best loss: 13.722250 Accuracy: 97.11% 9 Validation loss: 454.317535 Best loss: 13.722250 Accuracy: 97.54% 10 Validation loss: 1132.204102 Best loss: 13.722250 Accuracy: 96.05% 11 Validation loss: 503.987000 Best loss: 13.722250 Accuracy: 95.70% 12 Validation loss: 1752.538086 Best loss: 13.722250 Accuracy: 96.56% 13 Validation loss: 1163.239502 Best loss: 13.722250 Accuracy: 97.69% 14 Validation loss: 2752.798096 Best loss: 13.722250 Accuracy: 93.94% 15 Validation loss: 49518.425781 Best loss: 13.722250 Accuracy: 95.93% 16 Validation loss: 10567.247070 Best loss: 13.722250 Accuracy: 96.79% 17 Validation loss: 4139.558105 Best loss: 13.722250 Accuracy: 97.50% 18 Validation loss: 29931.947266 Best loss: 13.722250 Accuracy: 93.43% 19 Validation loss: 19089.832031 Best loss: 13.722250 Accuracy: 96.09% 20 Validation loss: 3997.277100 Best loss: 13.722250 Accuracy: 97.46% 21 Validation loss: 8915.182617 Best loss: 13.722250 Accuracy: 95.86% 22 Validation loss: 5469.671387 Best loss: 13.722250 Accuracy: 97.85% 23 Validation loss: 23844.810547 Best loss: 13.722250 Accuracy: 98.08% 24 Validation loss: 7908.539551 Best loss: 13.722250 Accuracy: 97.65% 25 Validation loss: 6178.676758 Best loss: 13.722250 Accuracy: 97.93% 26 Validation loss: 1774.994873 Best loss: 13.722250 Accuracy: 98.01% 27 Validation loss: 12466.194336 Best loss: 13.722250 Accuracy: 96.60% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=120, batch_size=10, learning_rate=0.02, total= 2.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 0.562190 Best loss: 0.562190 Accuracy: 87.53% 1 Validation loss: 0.196206 Best loss: 0.196206 Accuracy: 94.45% 2 Validation loss: 0.158882 Best loss: 0.158882 Accuracy: 95.19% 3 Validation loss: 0.136290 Best loss: 0.136290 Accuracy: 96.01% 4 Validation loss: 0.127035 Best loss: 0.127035 Accuracy: 96.29% 5 Validation loss: 0.136770 Best loss: 0.127035 Accuracy: 96.48% 6 Validation loss: 0.123501 Best loss: 0.123501 Accuracy: 96.83% 7 Validation loss: 0.110279 Best loss: 0.110279 Accuracy: 97.15% 8 Validation loss: 0.118504 Best loss: 0.110279 Accuracy: 97.50% 9 Validation loss: 0.149118 Best loss: 0.110279 Accuracy: 96.17% 10 Validation loss: 0.113160 Best loss: 0.110279 Accuracy: 97.34% 11 Validation loss: 0.104436 Best loss: 0.104436 Accuracy: 97.30% 12 Validation loss: 0.105355 Best loss: 0.104436 Accuracy: 97.22% 13 Validation loss: 0.094144 Best loss: 0.094144 Accuracy: 97.73% 14 Validation loss: 0.107384 Best loss: 0.094144 Accuracy: 97.77% 15 Validation loss: 0.099572 Best loss: 0.094144 Accuracy: 97.34% 16 Validation loss: 0.093942 Best loss: 0.093942 Accuracy: 97.69% 17 Validation loss: 0.111062 Best loss: 0.093942 Accuracy: 97.62% 18 Validation loss: 0.110193 Best loss: 0.093942 Accuracy: 97.34% 19 Validation loss: 0.115274 Best loss: 0.093942 Accuracy: 97.58% 20 Validation loss: 0.109975 Best loss: 0.093942 Accuracy: 97.38% 21 Validation loss: 0.107372 Best loss: 0.093942 Accuracy: 97.46% 22 Validation loss: 0.106285 Best loss: 0.093942 Accuracy: 97.77% 23 Validation loss: 0.104691 Best loss: 0.093942 Accuracy: 97.69% 24 Validation loss: 0.116564 Best loss: 0.093942 Accuracy: 97.34% 25 Validation loss: 0.107665 Best loss: 0.093942 Accuracy: 97.65% 26 Validation loss: 0.118755 Best loss: 0.093942 Accuracy: 97.65% 27 Validation loss: 0.094117 Best loss: 0.093942 Accuracy: 97.65% 28 Validation loss: 0.143342 Best loss: 0.093942 Accuracy: 97.50% 29 Validation loss: 0.127718 Best loss: 0.093942 Accuracy: 97.19% 30 Validation loss: 0.114440 Best loss: 0.093942 Accuracy: 97.58% 31 Validation loss: 0.114920 Best loss: 0.093942 Accuracy: 97.30% 32 Validation loss: 0.119774 Best loss: 0.093942 Accuracy: 97.93% 33 Validation loss: 0.118153 Best loss: 0.093942 Accuracy: 97.73% 34 Validation loss: 0.109703 Best loss: 0.093942 Accuracy: 97.54% 35 Validation loss: 0.337539 Best loss: 0.093942 Accuracy: 95.31% 36 Validation loss: 169.350739 Best loss: 0.093942 Accuracy: 22.13% 37 Validation loss: 2.308149 Best loss: 0.093942 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 4.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 0.794070 Best loss: 0.794070 Accuracy: 66.15% 1 Validation loss: 0.278878 Best loss: 0.278878 Accuracy: 92.49% 2 Validation loss: 0.174114 Best loss: 0.174114 Accuracy: 95.04% 3 Validation loss: 0.138463 Best loss: 0.138463 Accuracy: 95.86% 4 Validation loss: 0.154078 Best loss: 0.138463 Accuracy: 95.54% 5 Validation loss: 0.129906 Best loss: 0.129906 Accuracy: 96.13% 6 Validation loss: 0.112315 Best loss: 0.112315 Accuracy: 96.79% 7 Validation loss: 0.132121 Best loss: 0.112315 Accuracy: 96.52% 8 Validation loss: 0.121925 Best loss: 0.112315 Accuracy: 96.21% 9 Validation loss: 0.101795 Best loss: 0.101795 Accuracy: 96.99% 10 Validation loss: 0.106389 Best loss: 0.101795 Accuracy: 97.22% 11 Validation loss: 0.088449 Best loss: 0.088449 Accuracy: 97.62% 12 Validation loss: 0.123712 Best loss: 0.088449 Accuracy: 96.64% 13 Validation loss: 0.104357 Best loss: 0.088449 Accuracy: 97.03% 14 Validation loss: 0.086237 Best loss: 0.086237 Accuracy: 97.62% 15 Validation loss: 0.097118 Best loss: 0.086237 Accuracy: 97.62% 16 Validation loss: 0.073575 Best loss: 0.073575 Accuracy: 97.85% 17 Validation loss: 0.093589 Best loss: 0.073575 Accuracy: 97.77% 18 Validation loss: 0.078375 Best loss: 0.073575 Accuracy: 97.93% 19 Validation loss: 0.097763 Best loss: 0.073575 Accuracy: 97.30% 20 Validation loss: 0.093091 Best loss: 0.073575 Accuracy: 97.46% 21 Validation loss: 0.101676 Best loss: 0.073575 Accuracy: 97.22% 22 Validation loss: 0.095401 Best loss: 0.073575 Accuracy: 97.30% 23 Validation loss: 0.100575 Best loss: 0.073575 Accuracy: 97.54% 24 Validation loss: 0.081409 Best loss: 0.073575 Accuracy: 97.89% 25 Validation loss: 0.089388 Best loss: 0.073575 Accuracy: 97.85% 26 Validation loss: 0.089784 Best loss: 0.073575 Accuracy: 97.58% 27 Validation loss: 0.104534 Best loss: 0.073575 Accuracy: 97.54% 28 Validation loss: 0.118132 Best loss: 0.073575 Accuracy: 97.62% 29 Validation loss: 0.112633 Best loss: 0.073575 Accuracy: 97.62% 30 Validation loss: 0.103648 Best loss: 0.073575 Accuracy: 97.30% 31 Validation loss: 0.096751 Best loss: 0.073575 Accuracy: 98.05% 32 Validation loss: 0.117350 Best loss: 0.073575 Accuracy: 97.50% 33 Validation loss: 0.132025 Best loss: 0.073575 Accuracy: 97.77% 34 Validation loss: 0.132988 Best loss: 0.073575 Accuracy: 97.46% 35 Validation loss: 0.134534 Best loss: 0.073575 Accuracy: 97.85% 36 Validation loss: 0.112088 Best loss: 0.073575 Accuracy: 97.73% 37 Validation loss: 0.105948 Best loss: 0.073575 Accuracy: 97.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 4.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 1.018099 Best loss: 1.018099 Accuracy: 60.56% 1 Validation loss: 0.280845 Best loss: 0.280845 Accuracy: 91.63% 2 Validation loss: 0.180065 Best loss: 0.180065 Accuracy: 95.04% 3 Validation loss: 0.134649 Best loss: 0.134649 Accuracy: 96.21% 4 Validation loss: 0.131871 Best loss: 0.131871 Accuracy: 96.60% 5 Validation loss: 0.118228 Best loss: 0.118228 Accuracy: 96.99% 6 Validation loss: 0.113236 Best loss: 0.113236 Accuracy: 97.19% 7 Validation loss: 0.120056 Best loss: 0.113236 Accuracy: 96.44% 8 Validation loss: 0.111670 Best loss: 0.111670 Accuracy: 97.22% 9 Validation loss: 0.103891 Best loss: 0.103891 Accuracy: 97.26% 10 Validation loss: 0.100750 Best loss: 0.100750 Accuracy: 97.30% 11 Validation loss: 0.103473 Best loss: 0.100750 Accuracy: 97.38% 12 Validation loss: 0.119128 Best loss: 0.100750 Accuracy: 96.87% 13 Validation loss: 0.088451 Best loss: 0.088451 Accuracy: 97.65% 14 Validation loss: 0.105455 Best loss: 0.088451 Accuracy: 97.26% 15 Validation loss: 0.091730 Best loss: 0.088451 Accuracy: 97.46% 16 Validation loss: 0.118689 Best loss: 0.088451 Accuracy: 96.91% 17 Validation loss: 0.115736 Best loss: 0.088451 Accuracy: 97.19% 18 Validation loss: 0.091308 Best loss: 0.088451 Accuracy: 97.50% 19 Validation loss: 0.116531 Best loss: 0.088451 Accuracy: 97.42% 20 Validation loss: 0.102719 Best loss: 0.088451 Accuracy: 97.85% 21 Validation loss: 0.094270 Best loss: 0.088451 Accuracy: 97.54% 22 Validation loss: 0.100425 Best loss: 0.088451 Accuracy: 97.19% 23 Validation loss: 0.091521 Best loss: 0.088451 Accuracy: 97.54% 24 Validation loss: 0.101561 Best loss: 0.088451 Accuracy: 97.46% 25 Validation loss: 0.092087 Best loss: 0.088451 Accuracy: 97.77% 26 Validation loss: 0.110244 Best loss: 0.088451 Accuracy: 97.46% 27 Validation loss: 0.126442 Best loss: 0.088451 Accuracy: 97.07% 28 Validation loss: 0.204650 Best loss: 0.088451 Accuracy: 96.36% 29 Validation loss: 0.131054 Best loss: 0.088451 Accuracy: 96.99% 30 Validation loss: 0.144852 Best loss: 0.088451 Accuracy: 97.77% 31 Validation loss: 26781.144531 Best loss: 0.088451 Accuracy: 16.61% 32 Validation loss: 15.594292 Best loss: 0.088451 Accuracy: 22.01% 33 Validation loss: 2.236545 Best loss: 0.088451 Accuracy: 22.01% 34 Validation loss: 1.617507 Best loss: 0.088451 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 4.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1 0 Validation loss: 373908.531250 Best loss: 373908.531250 Accuracy: 19.19% 1 Validation loss: 608.462952 Best loss: 608.462952 Accuracy: 75.68% 2 Validation loss: 1368.843872 Best loss: 608.462952 Accuracy: 63.72% 3 Validation loss: 720.043823 Best loss: 608.462952 Accuracy: 78.19% 4 Validation loss: 298.639252 Best loss: 298.639252 Accuracy: 87.41% 5 Validation loss: 156.770355 Best loss: 156.770355 Accuracy: 91.87% 6 Validation loss: 146.501678 Best loss: 146.501678 Accuracy: 92.92% 7 Validation loss: 111.232140 Best loss: 111.232140 Accuracy: 93.82% 8 Validation loss: 362.617462 Best loss: 111.232140 Accuracy: 90.85% 9 Validation loss: 5786.559082 Best loss: 111.232140 Accuracy: 48.08% 10 Validation loss: 4227.552246 Best loss: 111.232140 Accuracy: 66.97% 11 Validation loss: 2769.626221 Best loss: 111.232140 Accuracy: 69.59% 12 Validation loss: 5177.689453 Best loss: 111.232140 Accuracy: 60.16% 13 Validation loss: 2720.686279 Best loss: 111.232140 Accuracy: 72.32% 14 Validation loss: 1036.290161 Best loss: 111.232140 Accuracy: 82.49% 15 Validation loss: 1207.869873 Best loss: 111.232140 Accuracy: 84.05% 16 Validation loss: 1455.944946 Best loss: 111.232140 Accuracy: 84.56% 17 Validation loss: 34838.273438 Best loss: 111.232140 Accuracy: 71.89% 18 Validation loss: 15103.136719 Best loss: 111.232140 Accuracy: 75.14% 19 Validation loss: 20116.744141 Best loss: 111.232140 Accuracy: 81.47% 20 Validation loss: 35261.781250 Best loss: 111.232140 Accuracy: 64.46% 21 Validation loss: 30729.685547 Best loss: 111.232140 Accuracy: 77.83% 22 Validation loss: 20698.914062 Best loss: 111.232140 Accuracy: 79.16% 23 Validation loss: 25817.201172 Best loss: 111.232140 Accuracy: 78.97% 24 Validation loss: 6417.388672 Best loss: 111.232140 Accuracy: 91.09% 25 Validation loss: 4417.124512 Best loss: 111.232140 Accuracy: 90.89% 26 Validation loss: 4464.074219 Best loss: 111.232140 Accuracy: 92.57% 27 Validation loss: 3104.923096 Best loss: 111.232140 Accuracy: 93.90% 28 Validation loss: 11024.225586 Best loss: 111.232140 Accuracy: 77.09% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1, total= 24.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1 0 Validation loss: 1.299679 Best loss: 1.299679 Accuracy: 45.50% 1 Validation loss: 1.014871 Best loss: 1.014871 Accuracy: 53.52% 2 Validation loss: 62573.953125 Best loss: 1.014871 Accuracy: 67.44% 3 Validation loss: 9562.051758 Best loss: 1.014871 Accuracy: 83.50% 4 Validation loss: 15153.731445 Best loss: 1.014871 Accuracy: 87.37% 5 Validation loss: 8348.821289 Best loss: 1.014871 Accuracy: 89.44% 6 Validation loss: 5524.603516 Best loss: 1.014871 Accuracy: 89.21% 7 Validation loss: 3770.559082 Best loss: 1.014871 Accuracy: 92.65% 8 Validation loss: 2276.135498 Best loss: 1.014871 Accuracy: 92.49% 9 Validation loss: 4838.749023 Best loss: 1.014871 Accuracy: 89.41% 10 Validation loss: 4605.408691 Best loss: 1.014871 Accuracy: 89.84% 11 Validation loss: 4071.097656 Best loss: 1.014871 Accuracy: 88.39% 12 Validation loss: 1703.522339 Best loss: 1.014871 Accuracy: 93.08% 13 Validation loss: 8942.528320 Best loss: 1.014871 Accuracy: 77.87% 14 Validation loss: 2065116.125000 Best loss: 1.014871 Accuracy: 18.73% 15 Validation loss: 99409.140625 Best loss: 1.014871 Accuracy: 91.13% 16 Validation loss: 69308.570312 Best loss: 1.014871 Accuracy: 77.64% 17 Validation loss: 23848.167969 Best loss: 1.014871 Accuracy: 95.50% 18 Validation loss: 28912.146484 Best loss: 1.014871 Accuracy: 95.78% 19 Validation loss: 18212.498047 Best loss: 1.014871 Accuracy: 93.00% 20 Validation loss: 26285.767578 Best loss: 1.014871 Accuracy: 81.98% 21 Validation loss: 12022.525391 Best loss: 1.014871 Accuracy: 96.40% 22 Validation loss: 18877.998047 Best loss: 1.014871 Accuracy: 90.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1, total= 21.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1 0 Validation loss: 554.976074 Best loss: 554.976074 Accuracy: 45.66% 1 Validation loss: 118.585968 Best loss: 118.585968 Accuracy: 59.62% 2 Validation loss: 25.711233 Best loss: 25.711233 Accuracy: 72.24% 3 Validation loss: 13.454128 Best loss: 13.454128 Accuracy: 84.83% 4 Validation loss: 967632.687500 Best loss: 13.454128 Accuracy: 26.54% 5 Validation loss: 144043.906250 Best loss: 13.454128 Accuracy: 54.22% 6 Validation loss: 29859.039062 Best loss: 13.454128 Accuracy: 71.93% 7 Validation loss: 30433.998047 Best loss: 13.454128 Accuracy: 69.55% 8 Validation loss: 39251.718750 Best loss: 13.454128 Accuracy: 70.99% 9 Validation loss: 36852.910156 Best loss: 13.454128 Accuracy: 80.18% 10 Validation loss: 56535.042969 Best loss: 13.454128 Accuracy: 65.32% 11 Validation loss: 20727.515625 Best loss: 13.454128 Accuracy: 75.88% 12 Validation loss: 14919.879883 Best loss: 13.454128 Accuracy: 82.53% 13 Validation loss: 8673.616211 Best loss: 13.454128 Accuracy: 86.75% 14 Validation loss: 17487.046875 Best loss: 13.454128 Accuracy: 85.34% 15 Validation loss: 5263.507324 Best loss: 13.454128 Accuracy: 89.48% 16 Validation loss: 6145.155762 Best loss: 13.454128 Accuracy: 89.05% 17 Validation loss: 14014.296875 Best loss: 13.454128 Accuracy: 76.90% 18 Validation loss: 5244.583496 Best loss: 13.454128 Accuracy: 91.16% 19 Validation loss: 8027.081543 Best loss: 13.454128 Accuracy: 78.58% 20 Validation loss: 3581.462158 Best loss: 13.454128 Accuracy: 89.48% 21 Validation loss: 2632.108398 Best loss: 13.454128 Accuracy: 91.05% 22 Validation loss: 2975.729736 Best loss: 13.454128 Accuracy: 90.62% 23 Validation loss: 1747.848877 Best loss: 13.454128 Accuracy: 92.22% 24 Validation loss: 3138.998047 Best loss: 13.454128 Accuracy: 82.76% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.1, total= 20.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1 0 Validation loss: 0.480153 Best loss: 0.480153 Accuracy: 92.06% 1 Validation loss: 0.421880 Best loss: 0.421880 Accuracy: 89.29% 2 Validation loss: 0.190869 Best loss: 0.190869 Accuracy: 95.19% 3 Validation loss: 0.187951 Best loss: 0.187951 Accuracy: 93.86% 4 Validation loss: 0.145558 Best loss: 0.145558 Accuracy: 96.25% 5 Validation loss: 0.174882 Best loss: 0.145558 Accuracy: 95.11% 6 Validation loss: 0.191565 Best loss: 0.145558 Accuracy: 94.21% 7 Validation loss: 0.151417 Best loss: 0.145558 Accuracy: 96.79% 8 Validation loss: 0.136837 Best loss: 0.136837 Accuracy: 96.72% 9 Validation loss: 0.180917 Best loss: 0.136837 Accuracy: 94.80% 10 Validation loss: 0.161785 Best loss: 0.136837 Accuracy: 96.87% 11 Validation loss: 0.129518 Best loss: 0.129518 Accuracy: 96.72% 12 Validation loss: 0.226618 Best loss: 0.129518 Accuracy: 96.95% 13 Validation loss: 0.144326 Best loss: 0.129518 Accuracy: 97.38% 14 Validation loss: 0.110794 Best loss: 0.110794 Accuracy: 97.03% 15 Validation loss: 0.307950 Best loss: 0.110794 Accuracy: 96.40% 16 Validation loss: 0.246384 Best loss: 0.110794 Accuracy: 97.03% 17 Validation loss: 2857709.500000 Best loss: 0.110794 Accuracy: 18.49% 18 Validation loss: 10350.208008 Best loss: 0.110794 Accuracy: 85.73% 19 Validation loss: 5866.630859 Best loss: 0.110794 Accuracy: 87.69% 20 Validation loss: 2811.059082 Best loss: 0.110794 Accuracy: 92.10% 21 Validation loss: 2877.493408 Best loss: 0.110794 Accuracy: 90.81% 22 Validation loss: 1906.922363 Best loss: 0.110794 Accuracy: 93.71% 23 Validation loss: 1831.168335 Best loss: 0.110794 Accuracy: 93.59% 24 Validation loss: 11896.711914 Best loss: 0.110794 Accuracy: 75.76% 25 Validation loss: 2894.582275 Best loss: 0.110794 Accuracy: 92.46% 26 Validation loss: 2618.649414 Best loss: 0.110794 Accuracy: 92.89% 27 Validation loss: 1542.160278 Best loss: 0.110794 Accuracy: 95.39% 28 Validation loss: 2201.673828 Best loss: 0.110794 Accuracy: 93.98% 29 Validation loss: 3604.220459 Best loss: 0.110794 Accuracy: 88.43% 30 Validation loss: 1693.232422 Best loss: 0.110794 Accuracy: 94.41% 31 Validation loss: 1211.298584 Best loss: 0.110794 Accuracy: 95.35% 32 Validation loss: 1575.235107 Best loss: 0.110794 Accuracy: 95.31% 33 Validation loss: 1649.490234 Best loss: 0.110794 Accuracy: 93.86% 34 Validation loss: 1652.113525 Best loss: 0.110794 Accuracy: 94.84% 35 Validation loss: 1105.714844 Best loss: 0.110794 Accuracy: 95.62% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1, total= 17.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1 0 Validation loss: 4493.494141 Best loss: 4493.494141 Accuracy: 59.27% 1 Validation loss: 191.505737 Best loss: 191.505737 Accuracy: 91.56% 2 Validation loss: 315.706451 Best loss: 191.505737 Accuracy: 93.63% 3 Validation loss: 169.856216 Best loss: 169.856216 Accuracy: 93.94% 4 Validation loss: 93.273888 Best loss: 93.273888 Accuracy: 94.06% 5 Validation loss: 144.807816 Best loss: 93.273888 Accuracy: 95.39% 6 Validation loss: 76.071747 Best loss: 76.071747 Accuracy: 95.39% 7 Validation loss: 58.506706 Best loss: 58.506706 Accuracy: 96.01% 8 Validation loss: 56.405323 Best loss: 56.405323 Accuracy: 94.49% 9 Validation loss: 35.002930 Best loss: 35.002930 Accuracy: 96.56% 10 Validation loss: 43.990387 Best loss: 35.002930 Accuracy: 96.48% 11 Validation loss: 123.268089 Best loss: 35.002930 Accuracy: 94.72% 12 Validation loss: 35.244652 Best loss: 35.002930 Accuracy: 96.87% 13 Validation loss: 30.364983 Best loss: 30.364983 Accuracy: 96.72% 14 Validation loss: 21.007750 Best loss: 21.007750 Accuracy: 96.79% 15 Validation loss: 1404.800903 Best loss: 21.007750 Accuracy: 77.05% 16 Validation loss: 58868.953125 Best loss: 21.007750 Accuracy: 78.81% 17 Validation loss: 21893.960938 Best loss: 21.007750 Accuracy: 88.31% 18 Validation loss: 16401.460938 Best loss: 21.007750 Accuracy: 91.59% 19 Validation loss: 29055.339844 Best loss: 21.007750 Accuracy: 83.85% 20 Validation loss: 16957.728516 Best loss: 21.007750 Accuracy: 92.26% 21 Validation loss: 14911.468750 Best loss: 21.007750 Accuracy: 91.05% 22 Validation loss: 27612.931641 Best loss: 21.007750 Accuracy: 88.12% 23 Validation loss: 9821.370117 Best loss: 21.007750 Accuracy: 93.55% 24 Validation loss: 15545.754883 Best loss: 21.007750 Accuracy: 90.38% 25 Validation loss: 23975.085938 Best loss: 21.007750 Accuracy: 92.85% 26 Validation loss: 31225.628906 Best loss: 21.007750 Accuracy: 86.43% 27 Validation loss: 8551.086914 Best loss: 21.007750 Accuracy: 95.07% 28 Validation loss: 7923.804688 Best loss: 21.007750 Accuracy: 95.11% 29 Validation loss: 20613.916016 Best loss: 21.007750 Accuracy: 85.46% 30 Validation loss: 12035.968750 Best loss: 21.007750 Accuracy: 94.45% 31 Validation loss: 19812.933594 Best loss: 21.007750 Accuracy: 92.57% 32 Validation loss: 6266.508789 Best loss: 21.007750 Accuracy: 95.11% 33 Validation loss: 11849.908203 Best loss: 21.007750 Accuracy: 91.20% 34 Validation loss: 20241.958984 Best loss: 21.007750 Accuracy: 90.54% 35 Validation loss: 9608.421875 Best loss: 21.007750 Accuracy: 95.35% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1, total= 17.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1 0 Validation loss: 0.777209 Best loss: 0.777209 Accuracy: 79.05% 1 Validation loss: 0.204957 Best loss: 0.204957 Accuracy: 94.64% 2 Validation loss: 0.190003 Best loss: 0.190003 Accuracy: 96.21% 3 Validation loss: 0.706049 Best loss: 0.190003 Accuracy: 93.71% 4 Validation loss: 0.125459 Best loss: 0.125459 Accuracy: 97.11% 5 Validation loss: 0.122097 Best loss: 0.122097 Accuracy: 96.52% 6 Validation loss: 0.125469 Best loss: 0.122097 Accuracy: 96.05% 7 Validation loss: 0.124558 Best loss: 0.122097 Accuracy: 96.99% 8 Validation loss: 0.135355 Best loss: 0.122097 Accuracy: 97.19% 9 Validation loss: 0.130055 Best loss: 0.122097 Accuracy: 96.72% 10 Validation loss: 0.221720 Best loss: 0.122097 Accuracy: 94.41% 11 Validation loss: 0.092869 Best loss: 0.092869 Accuracy: 97.58% 12 Validation loss: 0.181106 Best loss: 0.092869 Accuracy: 96.40% 13 Validation loss: 0.095960 Best loss: 0.092869 Accuracy: 97.50% 14 Validation loss: 378047.593750 Best loss: 0.092869 Accuracy: 50.16% 15 Validation loss: 33166.011719 Best loss: 0.092869 Accuracy: 82.21% 16 Validation loss: 29013.720703 Best loss: 0.092869 Accuracy: 83.97% 17 Validation loss: 50589.929688 Best loss: 0.092869 Accuracy: 73.34% 18 Validation loss: 21050.390625 Best loss: 0.092869 Accuracy: 85.14% 19 Validation loss: 12107.888672 Best loss: 0.092869 Accuracy: 92.34% 20 Validation loss: 9849.708984 Best loss: 0.092869 Accuracy: 93.16% 21 Validation loss: 14809.724609 Best loss: 0.092869 Accuracy: 89.21% 22 Validation loss: 5561.022461 Best loss: 0.092869 Accuracy: 94.64% 23 Validation loss: 7122.050781 Best loss: 0.092869 Accuracy: 94.72% 24 Validation loss: 7649.153809 Best loss: 0.092869 Accuracy: 93.67% 25 Validation loss: 15467.032227 Best loss: 0.092869 Accuracy: 91.67% 26 Validation loss: 8609.334961 Best loss: 0.092869 Accuracy: 93.67% 27 Validation loss: 6122.954590 Best loss: 0.092869 Accuracy: 94.49% 28 Validation loss: 5223.626465 Best loss: 0.092869 Accuracy: 95.93% 29 Validation loss: 7824.312012 Best loss: 0.092869 Accuracy: 93.67% 30 Validation loss: 19139.787109 Best loss: 0.092869 Accuracy: 88.55% 31 Validation loss: 5059.013672 Best loss: 0.092869 Accuracy: 95.31% 32 Validation loss: 19361.988281 Best loss: 0.092869 Accuracy: 87.14% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.1, total= 15.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05 0 Validation loss: 2.120880 Best loss: 2.120880 Accuracy: 22.01% 1 Validation loss: 2.412750 Best loss: 2.120880 Accuracy: 18.73% 2 Validation loss: 2.032418 Best loss: 2.032418 Accuracy: 19.27% 3 Validation loss: 2.181904 Best loss: 2.032418 Accuracy: 19.27% 4 Validation loss: 2.031989 Best loss: 2.031989 Accuracy: 19.27% 5 Validation loss: 1.686523 Best loss: 1.686523 Accuracy: 20.91% 6 Validation loss: 2.098040 Best loss: 1.686523 Accuracy: 18.73% 7 Validation loss: 1.991273 Best loss: 1.686523 Accuracy: 19.08% 8 Validation loss: 1.886133 Best loss: 1.686523 Accuracy: 19.08% 9 Validation loss: 2.482888 Best loss: 1.686523 Accuracy: 19.27% 10 Validation loss: 1.726855 Best loss: 1.686523 Accuracy: 20.91% 11 Validation loss: 1.677547 Best loss: 1.677547 Accuracy: 22.01% 12 Validation loss: 1.999843 Best loss: 1.677547 Accuracy: 19.08% 13 Validation loss: 1.756567 Best loss: 1.677547 Accuracy: 19.08% 14 Validation loss: 2.995759 Best loss: 1.677547 Accuracy: 20.91% 15 Validation loss: 1.696039 Best loss: 1.677547 Accuracy: 19.27% 16 Validation loss: 2.644183 Best loss: 1.677547 Accuracy: 20.91% 17 Validation loss: 2.019509 Best loss: 1.677547 Accuracy: 20.91% 18 Validation loss: 2.568128 Best loss: 1.677547 Accuracy: 22.01% 19 Validation loss: 2.833771 Best loss: 1.677547 Accuracy: 22.01% 20 Validation loss: 2.283546 Best loss: 1.677547 Accuracy: 22.01% 21 Validation loss: 2.377193 Best loss: 1.677547 Accuracy: 20.91% 22 Validation loss: 1.983351 Best loss: 1.677547 Accuracy: 18.73% 23 Validation loss: 2.143963 Best loss: 1.677547 Accuracy: 19.27% 24 Validation loss: 2.032374 Best loss: 1.677547 Accuracy: 20.91% 25 Validation loss: 1.827981 Best loss: 1.677547 Accuracy: 20.91% 26 Validation loss: 1.893215 Best loss: 1.677547 Accuracy: 19.08% 27 Validation loss: 2.163725 Best loss: 1.677547 Accuracy: 22.01% 28 Validation loss: 2.621582 Best loss: 1.677547 Accuracy: 19.08% 29 Validation loss: 2.281033 Best loss: 1.677547 Accuracy: 20.91% 30 Validation loss: 1.913919 Best loss: 1.677547 Accuracy: 19.27% 31 Validation loss: 2.141626 Best loss: 1.677547 Accuracy: 19.27% 32 Validation loss: 2.017712 Best loss: 1.677547 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05, total= 1.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05 0 Validation loss: 1.876559 Best loss: 1.876559 Accuracy: 18.73% 1 Validation loss: 2.085444 Best loss: 1.876559 Accuracy: 19.27% 2 Validation loss: 2.172938 Best loss: 1.876559 Accuracy: 22.01% 3 Validation loss: 2.179469 Best loss: 1.876559 Accuracy: 22.01% 4 Validation loss: 2.008929 Best loss: 1.876559 Accuracy: 20.91% 5 Validation loss: 2.124995 Best loss: 1.876559 Accuracy: 19.27% 6 Validation loss: 1.798582 Best loss: 1.798582 Accuracy: 19.27% 7 Validation loss: 2.029090 Best loss: 1.798582 Accuracy: 19.08% 8 Validation loss: 2.358320 Best loss: 1.798582 Accuracy: 18.73% 9 Validation loss: 1.772398 Best loss: 1.772398 Accuracy: 19.27% 10 Validation loss: 2.146658 Best loss: 1.772398 Accuracy: 19.08% 11 Validation loss: 2.742315 Best loss: 1.772398 Accuracy: 22.01% 12 Validation loss: 2.136126 Best loss: 1.772398 Accuracy: 19.27% 13 Validation loss: 2.504884 Best loss: 1.772398 Accuracy: 19.27% 14 Validation loss: 2.295611 Best loss: 1.772398 Accuracy: 19.27% 15 Validation loss: 2.066975 Best loss: 1.772398 Accuracy: 18.73% 16 Validation loss: 1.657117 Best loss: 1.657117 Accuracy: 19.08% 17 Validation loss: 1.928222 Best loss: 1.657117 Accuracy: 18.73% 18 Validation loss: 1.869563 Best loss: 1.657117 Accuracy: 19.08% 19 Validation loss: 2.003770 Best loss: 1.657117 Accuracy: 18.73% 20 Validation loss: 2.406114 Best loss: 1.657117 Accuracy: 19.27% 21 Validation loss: 1.857649 Best loss: 1.657117 Accuracy: 19.27% 22 Validation loss: 2.096572 Best loss: 1.657117 Accuracy: 19.27% 23 Validation loss: 2.520547 Best loss: 1.657117 Accuracy: 22.01% 24 Validation loss: 1.893651 Best loss: 1.657117 Accuracy: 19.08% 25 Validation loss: 2.078128 Best loss: 1.657117 Accuracy: 22.01% 26 Validation loss: 2.804218 Best loss: 1.657117 Accuracy: 19.08% 27 Validation loss: 2.394260 Best loss: 1.657117 Accuracy: 20.91% 28 Validation loss: 1.977028 Best loss: 1.657117 Accuracy: 20.91% 29 Validation loss: 1.721668 Best loss: 1.657117 Accuracy: 19.27% 30 Validation loss: 2.204547 Best loss: 1.657117 Accuracy: 19.08% 31 Validation loss: 2.605132 Best loss: 1.657117 Accuracy: 22.01% 32 Validation loss: 2.066092 Best loss: 1.657117 Accuracy: 18.73% 33 Validation loss: 2.527795 Best loss: 1.657117 Accuracy: 19.08% 34 Validation loss: 2.730561 Best loss: 1.657117 Accuracy: 18.73% 35 Validation loss: 2.582782 Best loss: 1.657117 Accuracy: 22.01% 36 Validation loss: 2.845665 Best loss: 1.657117 Accuracy: 19.27% 37 Validation loss: 1.998368 Best loss: 1.657117 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05, total= 2.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05 0 Validation loss: 1.835345 Best loss: 1.835345 Accuracy: 22.01% 1 Validation loss: 2.159543 Best loss: 1.835345 Accuracy: 22.01% 2 Validation loss: 2.201642 Best loss: 1.835345 Accuracy: 19.08% 3 Validation loss: 2.145575 Best loss: 1.835345 Accuracy: 19.27% 4 Validation loss: 3.100879 Best loss: 1.835345 Accuracy: 20.91% 5 Validation loss: 2.464484 Best loss: 1.835345 Accuracy: 20.91% 6 Validation loss: 2.003295 Best loss: 1.835345 Accuracy: 22.01% 7 Validation loss: 1.763759 Best loss: 1.763759 Accuracy: 22.01% 8 Validation loss: 1.700588 Best loss: 1.700588 Accuracy: 19.27% 9 Validation loss: 2.334472 Best loss: 1.700588 Accuracy: 22.01% 10 Validation loss: 1.853941 Best loss: 1.700588 Accuracy: 18.73% 11 Validation loss: 1.842175 Best loss: 1.700588 Accuracy: 19.27% 12 Validation loss: 2.091004 Best loss: 1.700588 Accuracy: 20.91% 13 Validation loss: 3.280350 Best loss: 1.700588 Accuracy: 19.08% 14 Validation loss: 2.147485 Best loss: 1.700588 Accuracy: 19.08% 15 Validation loss: 1.816309 Best loss: 1.700588 Accuracy: 19.27% 16 Validation loss: 1.802174 Best loss: 1.700588 Accuracy: 18.73% 17 Validation loss: 2.435385 Best loss: 1.700588 Accuracy: 20.91% 18 Validation loss: 1.970502 Best loss: 1.700588 Accuracy: 19.27% 19 Validation loss: 2.368727 Best loss: 1.700588 Accuracy: 22.01% 20 Validation loss: 2.211079 Best loss: 1.700588 Accuracy: 19.27% 21 Validation loss: 2.549935 Best loss: 1.700588 Accuracy: 22.01% 22 Validation loss: 1.926468 Best loss: 1.700588 Accuracy: 19.27% 23 Validation loss: 2.673764 Best loss: 1.700588 Accuracy: 19.08% 24 Validation loss: 2.478780 Best loss: 1.700588 Accuracy: 19.27% 25 Validation loss: 1.703409 Best loss: 1.700588 Accuracy: 22.01% 26 Validation loss: 2.281162 Best loss: 1.700588 Accuracy: 18.73% 27 Validation loss: 1.997834 Best loss: 1.700588 Accuracy: 18.73% 28 Validation loss: 2.451638 Best loss: 1.700588 Accuracy: 19.08% 29 Validation loss: 2.782292 Best loss: 1.700588 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.05, total= 1.5min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02 0 Validation loss: 0.136604 Best loss: 0.136604 Accuracy: 95.39% 1 Validation loss: 0.085477 Best loss: 0.085477 Accuracy: 97.34% 2 Validation loss: 0.075440 Best loss: 0.075440 Accuracy: 97.73% 3 Validation loss: 0.085036 Best loss: 0.075440 Accuracy: 97.19% 4 Validation loss: 0.058059 Best loss: 0.058059 Accuracy: 98.48% 5 Validation loss: 0.060840 Best loss: 0.058059 Accuracy: 98.08% 6 Validation loss: 0.057152 Best loss: 0.057152 Accuracy: 98.40% 7 Validation loss: 0.076500 Best loss: 0.057152 Accuracy: 98.20% 8 Validation loss: 0.053095 Best loss: 0.053095 Accuracy: 98.63% 9 Validation loss: 0.048657 Best loss: 0.048657 Accuracy: 98.83% 10 Validation loss: 0.058482 Best loss: 0.048657 Accuracy: 98.79% 11 Validation loss: 0.056887 Best loss: 0.048657 Accuracy: 98.75% 12 Validation loss: 0.071831 Best loss: 0.048657 Accuracy: 98.28% 13 Validation loss: 0.045360 Best loss: 0.045360 Accuracy: 98.87% 14 Validation loss: 0.056967 Best loss: 0.045360 Accuracy: 98.48% 15 Validation loss: 0.067861 Best loss: 0.045360 Accuracy: 98.36% 16 Validation loss: 0.065811 Best loss: 0.045360 Accuracy: 98.67% 17 Validation loss: 0.054928 Best loss: 0.045360 Accuracy: 98.94% 18 Validation loss: 0.071558 Best loss: 0.045360 Accuracy: 98.75% 19 Validation loss: 0.065895 Best loss: 0.045360 Accuracy: 98.55% 20 Validation loss: 0.072200 Best loss: 0.045360 Accuracy: 98.51% 21 Validation loss: 0.077181 Best loss: 0.045360 Accuracy: 98.83% 22 Validation loss: 0.061244 Best loss: 0.045360 Accuracy: 98.71% 23 Validation loss: 0.081296 Best loss: 0.045360 Accuracy: 98.63% 24 Validation loss: 0.078714 Best loss: 0.045360 Accuracy: 98.32% 25 Validation loss: 0.074440 Best loss: 0.045360 Accuracy: 98.24% 26 Validation loss: 0.058888 Best loss: 0.045360 Accuracy: 98.67% 27 Validation loss: 0.076918 Best loss: 0.045360 Accuracy: 98.75% 28 Validation loss: 0.067392 Best loss: 0.045360 Accuracy: 98.59% 29 Validation loss: 0.067202 Best loss: 0.045360 Accuracy: 98.67% 30 Validation loss: 0.069936 Best loss: 0.045360 Accuracy: 98.94% 31 Validation loss: 0.069238 Best loss: 0.045360 Accuracy: 98.79% 32 Validation loss: 0.068983 Best loss: 0.045360 Accuracy: 98.79% 33 Validation loss: 0.055486 Best loss: 0.045360 Accuracy: 98.75% 34 Validation loss: 0.081158 Best loss: 0.045360 Accuracy: 98.55% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02, total= 3.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02 0 Validation loss: 0.173685 Best loss: 0.173685 Accuracy: 95.19% 1 Validation loss: 0.088298 Best loss: 0.088298 Accuracy: 97.30% 2 Validation loss: 0.069179 Best loss: 0.069179 Accuracy: 97.81% 3 Validation loss: 0.069277 Best loss: 0.069179 Accuracy: 97.97% 4 Validation loss: 0.064584 Best loss: 0.064584 Accuracy: 98.05% 5 Validation loss: 0.077268 Best loss: 0.064584 Accuracy: 97.89% 6 Validation loss: 0.056205 Best loss: 0.056205 Accuracy: 98.20% 7 Validation loss: 0.062621 Best loss: 0.056205 Accuracy: 98.28% 8 Validation loss: 0.066903 Best loss: 0.056205 Accuracy: 98.40% 9 Validation loss: 0.081417 Best loss: 0.056205 Accuracy: 98.24% 10 Validation loss: 0.064616 Best loss: 0.056205 Accuracy: 98.36% 11 Validation loss: 0.055205 Best loss: 0.055205 Accuracy: 98.71% 12 Validation loss: 0.062418 Best loss: 0.055205 Accuracy: 98.67% 13 Validation loss: 0.070713 Best loss: 0.055205 Accuracy: 98.55% 14 Validation loss: 0.080402 Best loss: 0.055205 Accuracy: 98.24% 15 Validation loss: 0.064902 Best loss: 0.055205 Accuracy: 98.91% 16 Validation loss: 0.087787 Best loss: 0.055205 Accuracy: 98.36% 17 Validation loss: 0.093843 Best loss: 0.055205 Accuracy: 98.40% 18 Validation loss: 0.062505 Best loss: 0.055205 Accuracy: 98.67% 19 Validation loss: 0.049314 Best loss: 0.049314 Accuracy: 99.06% 20 Validation loss: 0.061369 Best loss: 0.049314 Accuracy: 98.63% 21 Validation loss: 0.062460 Best loss: 0.049314 Accuracy: 98.71% 22 Validation loss: 0.059812 Best loss: 0.049314 Accuracy: 98.87% 23 Validation loss: 0.074500 Best loss: 0.049314 Accuracy: 98.40% 24 Validation loss: 0.072534 Best loss: 0.049314 Accuracy: 98.67% 25 Validation loss: 0.081171 Best loss: 0.049314 Accuracy: 98.44% 26 Validation loss: 0.073866 Best loss: 0.049314 Accuracy: 98.36% 27 Validation loss: 0.063368 Best loss: 0.049314 Accuracy: 98.48% 28 Validation loss: 0.072661 Best loss: 0.049314 Accuracy: 98.75% 29 Validation loss: 0.074902 Best loss: 0.049314 Accuracy: 98.75% 30 Validation loss: 0.056950 Best loss: 0.049314 Accuracy: 98.98% 31 Validation loss: 0.084532 Best loss: 0.049314 Accuracy: 98.40% 32 Validation loss: 0.088277 Best loss: 0.049314 Accuracy: 98.91% 33 Validation loss: 0.085589 Best loss: 0.049314 Accuracy: 98.87% 34 Validation loss: 0.065951 Best loss: 0.049314 Accuracy: 98.87% 35 Validation loss: 0.069434 Best loss: 0.049314 Accuracy: 98.83% 36 Validation loss: 0.071008 Best loss: 0.049314 Accuracy: 98.79% 37 Validation loss: 0.084294 Best loss: 0.049314 Accuracy: 98.63% 38 Validation loss: 0.053079 Best loss: 0.049314 Accuracy: 98.87% 39 Validation loss: 0.091408 Best loss: 0.049314 Accuracy: 98.87% 40 Validation loss: 0.064609 Best loss: 0.049314 Accuracy: 99.02% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02, total= 4.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02 0 Validation loss: 0.132825 Best loss: 0.132825 Accuracy: 95.93% 1 Validation loss: 0.097318 Best loss: 0.097318 Accuracy: 97.07% 2 Validation loss: 0.080755 Best loss: 0.080755 Accuracy: 97.81% 3 Validation loss: 0.068622 Best loss: 0.068622 Accuracy: 97.97% 4 Validation loss: 0.065148 Best loss: 0.065148 Accuracy: 98.12% 5 Validation loss: 0.060857 Best loss: 0.060857 Accuracy: 98.08% 6 Validation loss: 0.067322 Best loss: 0.060857 Accuracy: 97.93% 7 Validation loss: 0.053426 Best loss: 0.053426 Accuracy: 98.32% 8 Validation loss: 0.052743 Best loss: 0.052743 Accuracy: 98.59% 9 Validation loss: 0.046836 Best loss: 0.046836 Accuracy: 98.87% 10 Validation loss: 0.051211 Best loss: 0.046836 Accuracy: 98.51% 11 Validation loss: 0.072841 Best loss: 0.046836 Accuracy: 98.32% 12 Validation loss: 0.052570 Best loss: 0.046836 Accuracy: 98.59% 13 Validation loss: 0.057716 Best loss: 0.046836 Accuracy: 98.71% 14 Validation loss: 0.093526 Best loss: 0.046836 Accuracy: 97.81% 15 Validation loss: 0.046600 Best loss: 0.046600 Accuracy: 98.71% 16 Validation loss: 0.057055 Best loss: 0.046600 Accuracy: 98.67% 17 Validation loss: 0.052680 Best loss: 0.046600 Accuracy: 98.75% 18 Validation loss: 0.064373 Best loss: 0.046600 Accuracy: 98.75% 19 Validation loss: 0.055292 Best loss: 0.046600 Accuracy: 98.71% 20 Validation loss: 0.077645 Best loss: 0.046600 Accuracy: 98.71% 21 Validation loss: 0.073124 Best loss: 0.046600 Accuracy: 98.55% 22 Validation loss: 0.071070 Best loss: 0.046600 Accuracy: 98.63% 23 Validation loss: 0.066862 Best loss: 0.046600 Accuracy: 98.63% 24 Validation loss: 0.081224 Best loss: 0.046600 Accuracy: 98.63% 25 Validation loss: 0.133893 Best loss: 0.046600 Accuracy: 98.08% 26 Validation loss: 0.081867 Best loss: 0.046600 Accuracy: 98.63% 27 Validation loss: 0.066053 Best loss: 0.046600 Accuracy: 98.51% 28 Validation loss: 0.065882 Best loss: 0.046600 Accuracy: 98.91% 29 Validation loss: 0.072662 Best loss: 0.046600 Accuracy: 98.67% 30 Validation loss: 0.062813 Best loss: 0.046600 Accuracy: 98.91% 31 Validation loss: 0.075871 Best loss: 0.046600 Accuracy: 98.67% 32 Validation loss: 0.049903 Best loss: 0.046600 Accuracy: 98.83% 33 Validation loss: 0.077020 Best loss: 0.046600 Accuracy: 98.59% 34 Validation loss: 0.062460 Best loss: 0.046600 Accuracy: 98.91% 35 Validation loss: 0.048084 Best loss: 0.046600 Accuracy: 98.91% 36 Validation loss: 0.063239 Best loss: 0.046600 Accuracy: 98.94% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=500, learning_rate=0.02, total= 3.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05 0 Validation loss: 0.489212 Best loss: 0.489212 Accuracy: 80.41% 1 Validation loss: 0.185602 Best loss: 0.185602 Accuracy: 95.50% 2 Validation loss: 0.236737 Best loss: 0.185602 Accuracy: 94.88% 3 Validation loss: 0.238084 Best loss: 0.185602 Accuracy: 94.37% 4 Validation loss: 0.391449 Best loss: 0.185602 Accuracy: 89.25% 5 Validation loss: 0.450904 Best loss: 0.185602 Accuracy: 88.86% 6 Validation loss: 0.604302 Best loss: 0.185602 Accuracy: 72.32% 7 Validation loss: 1.110277 Best loss: 0.185602 Accuracy: 61.77% 8 Validation loss: 0.473778 Best loss: 0.185602 Accuracy: 86.32% 9 Validation loss: 0.717169 Best loss: 0.185602 Accuracy: 74.67% 10 Validation loss: 0.675704 Best loss: 0.185602 Accuracy: 75.96% 11 Validation loss: 0.773403 Best loss: 0.185602 Accuracy: 68.26% 12 Validation loss: 0.608551 Best loss: 0.185602 Accuracy: 77.21% 13 Validation loss: 0.545896 Best loss: 0.185602 Accuracy: 78.77% 14 Validation loss: 0.579575 Best loss: 0.185602 Accuracy: 77.44% 15 Validation loss: 0.573971 Best loss: 0.185602 Accuracy: 77.37% 16 Validation loss: 0.778242 Best loss: 0.185602 Accuracy: 70.41% 17 Validation loss: 0.920835 Best loss: 0.185602 Accuracy: 57.78% 18 Validation loss: 0.736933 Best loss: 0.185602 Accuracy: 73.42% 19 Validation loss: 0.866608 Best loss: 0.185602 Accuracy: 65.95% 20 Validation loss: 0.607533 Best loss: 0.185602 Accuracy: 76.35% 21 Validation loss: 0.664871 Best loss: 0.185602 Accuracy: 78.15% 22 Validation loss: 0.648947 Best loss: 0.185602 Accuracy: 77.09% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05, total= 11.9s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05 0 Validation loss: 1.015742 Best loss: 1.015742 Accuracy: 91.40% 1 Validation loss: 0.485136 Best loss: 0.485136 Accuracy: 84.48% 2 Validation loss: 0.359793 Best loss: 0.359793 Accuracy: 89.76% 3 Validation loss: 0.544737 Best loss: 0.359793 Accuracy: 77.52% 4 Validation loss: 0.644529 Best loss: 0.359793 Accuracy: 72.79% 5 Validation loss: 0.721121 Best loss: 0.359793 Accuracy: 60.48% 6 Validation loss: 0.799878 Best loss: 0.359793 Accuracy: 62.35% 7 Validation loss: 0.652892 Best loss: 0.359793 Accuracy: 73.57% 8 Validation loss: 0.883456 Best loss: 0.359793 Accuracy: 65.13% 9 Validation loss: 0.782492 Best loss: 0.359793 Accuracy: 67.20% 10 Validation loss: 0.804628 Best loss: 0.359793 Accuracy: 66.03% 11 Validation loss: 0.715618 Best loss: 0.359793 Accuracy: 68.80% 12 Validation loss: 1.100762 Best loss: 0.359793 Accuracy: 48.75% 13 Validation loss: 1.283635 Best loss: 0.359793 Accuracy: 35.34% 14 Validation loss: 1.278465 Best loss: 0.359793 Accuracy: 39.21% 15 Validation loss: 1.273548 Best loss: 0.359793 Accuracy: 39.21% 16 Validation loss: 1.281013 Best loss: 0.359793 Accuracy: 39.21% 17 Validation loss: 1.276124 Best loss: 0.359793 Accuracy: 38.35% 18 Validation loss: 1.276854 Best loss: 0.359793 Accuracy: 39.21% 19 Validation loss: 1.271194 Best loss: 0.359793 Accuracy: 39.21% 20 Validation loss: 1.271496 Best loss: 0.359793 Accuracy: 39.21% 21 Validation loss: 1.286627 Best loss: 0.359793 Accuracy: 39.21% 22 Validation loss: 1.285125 Best loss: 0.359793 Accuracy: 39.21% 23 Validation loss: 1.284192 Best loss: 0.359793 Accuracy: 39.21% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05, total= 14.9s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05 0 Validation loss: 0.225215 Best loss: 0.225215 Accuracy: 95.74% 1 Validation loss: 0.681471 Best loss: 0.225215 Accuracy: 68.10% 2 Validation loss: 0.716228 Best loss: 0.225215 Accuracy: 70.68% 3 Validation loss: 0.598349 Best loss: 0.225215 Accuracy: 76.39% 4 Validation loss: 0.565729 Best loss: 0.225215 Accuracy: 84.40% 5 Validation loss: 0.573261 Best loss: 0.225215 Accuracy: 84.60% 6 Validation loss: 0.483589 Best loss: 0.225215 Accuracy: 85.46% 7 Validation loss: 0.515181 Best loss: 0.225215 Accuracy: 83.03% 8 Validation loss: 0.474680 Best loss: 0.225215 Accuracy: 84.68% 9 Validation loss: 0.443761 Best loss: 0.225215 Accuracy: 84.32% 10 Validation loss: 0.390158 Best loss: 0.225215 Accuracy: 89.33% 11 Validation loss: 0.411879 Best loss: 0.225215 Accuracy: 89.95% 12 Validation loss: 0.446163 Best loss: 0.225215 Accuracy: 86.90% 13 Validation loss: 0.427359 Best loss: 0.225215 Accuracy: 88.82% 14 Validation loss: 0.492195 Best loss: 0.225215 Accuracy: 88.98% 15 Validation loss: 0.454095 Best loss: 0.225215 Accuracy: 87.96% 16 Validation loss: 0.444898 Best loss: 0.225215 Accuracy: 89.87% 17 Validation loss: 0.402905 Best loss: 0.225215 Accuracy: 88.23% 18 Validation loss: 0.411593 Best loss: 0.225215 Accuracy: 90.15% 19 Validation loss: 0.543693 Best loss: 0.225215 Accuracy: 90.93% 20 Validation loss: 0.428698 Best loss: 0.225215 Accuracy: 89.80% 21 Validation loss: 1.223304 Best loss: 0.225215 Accuracy: 70.13% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=50, learning_rate=0.05, total= 14.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 0.146132 Best loss: 0.146132 Accuracy: 96.05% 1 Validation loss: 0.145392 Best loss: 0.145392 Accuracy: 95.31% 2 Validation loss: 1592.792969 Best loss: 0.145392 Accuracy: 19.08% 3 Validation loss: 26.749559 Best loss: 0.145392 Accuracy: 75.61% 4 Validation loss: 5.246194 Best loss: 0.145392 Accuracy: 88.98% 5 Validation loss: 5.217668 Best loss: 0.145392 Accuracy: 89.05% 6 Validation loss: 4.506704 Best loss: 0.145392 Accuracy: 89.60% 7 Validation loss: 3.872263 Best loss: 0.145392 Accuracy: 89.76% 8 Validation loss: 2.468026 Best loss: 0.145392 Accuracy: 93.04% 9 Validation loss: 2.102195 Best loss: 0.145392 Accuracy: 93.08% 10 Validation loss: 2.954633 Best loss: 0.145392 Accuracy: 92.34% 11 Validation loss: 3.469189 Best loss: 0.145392 Accuracy: 91.13% 12 Validation loss: 1.046432 Best loss: 0.145392 Accuracy: 95.11% 13 Validation loss: 1.388578 Best loss: 0.145392 Accuracy: 94.64% 14 Validation loss: 4.095583 Best loss: 0.145392 Accuracy: 89.84% 15 Validation loss: 2.183164 Best loss: 0.145392 Accuracy: 92.96% 16 Validation loss: 2.435832 Best loss: 0.145392 Accuracy: 95.15% 17 Validation loss: 1.374541 Best loss: 0.145392 Accuracy: 95.27% 18 Validation loss: 1.276761 Best loss: 0.145392 Accuracy: 95.43% 19 Validation loss: 1.066249 Best loss: 0.145392 Accuracy: 95.23% 20 Validation loss: 2.500587 Best loss: 0.145392 Accuracy: 87.33% 21 Validation loss: 1.610008 Best loss: 0.145392 Accuracy: 96.52% 22 Validation loss: 1.426327 Best loss: 0.145392 Accuracy: 95.43% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 9.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 0.143698 Best loss: 0.143698 Accuracy: 96.21% 1 Validation loss: 0.114154 Best loss: 0.114154 Accuracy: 96.44% 2 Validation loss: 9535.852539 Best loss: 0.114154 Accuracy: 18.73% 3 Validation loss: 10.360773 Best loss: 0.114154 Accuracy: 83.39% 4 Validation loss: 8.825236 Best loss: 0.114154 Accuracy: 84.01% 5 Validation loss: 3.162607 Best loss: 0.114154 Accuracy: 87.65% 6 Validation loss: 2.355881 Best loss: 0.114154 Accuracy: 90.70% 7 Validation loss: 2.026160 Best loss: 0.114154 Accuracy: 91.56% 8 Validation loss: 2.210745 Best loss: 0.114154 Accuracy: 89.33% 9 Validation loss: 1.792705 Best loss: 0.114154 Accuracy: 93.67% 10 Validation loss: 1.515435 Best loss: 0.114154 Accuracy: 92.06% 11 Validation loss: 1.053782 Best loss: 0.114154 Accuracy: 93.75% 12 Validation loss: 1.063251 Best loss: 0.114154 Accuracy: 94.49% 13 Validation loss: 1.423369 Best loss: 0.114154 Accuracy: 92.89% 14 Validation loss: 1.338648 Best loss: 0.114154 Accuracy: 93.35% 15 Validation loss: 1.045516 Best loss: 0.114154 Accuracy: 93.94% 16 Validation loss: 0.970758 Best loss: 0.114154 Accuracy: 95.23% 17 Validation loss: 0.998510 Best loss: 0.114154 Accuracy: 94.61% 18 Validation loss: 1.075223 Best loss: 0.114154 Accuracy: 94.84% 19 Validation loss: 2.470094 Best loss: 0.114154 Accuracy: 94.88% 20 Validation loss: 1.332176 Best loss: 0.114154 Accuracy: 95.90% 21 Validation loss: 2.149577 Best loss: 0.114154 Accuracy: 94.88% 22 Validation loss: 3.552390 Best loss: 0.114154 Accuracy: 94.76% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 11.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05 0 Validation loss: 0.167633 Best loss: 0.167633 Accuracy: 96.56% 1 Validation loss: 0.176932 Best loss: 0.167633 Accuracy: 95.97% 2 Validation loss: 0.111417 Best loss: 0.111417 Accuracy: 97.46% 3 Validation loss: 59.458649 Best loss: 0.111417 Accuracy: 71.81% 4 Validation loss: 19.745806 Best loss: 0.111417 Accuracy: 85.50% 5 Validation loss: 8.469905 Best loss: 0.111417 Accuracy: 90.30% 6 Validation loss: 5.247195 Best loss: 0.111417 Accuracy: 93.71% 7 Validation loss: 6.262508 Best loss: 0.111417 Accuracy: 91.71% 8 Validation loss: 10.424242 Best loss: 0.111417 Accuracy: 83.62% 9 Validation loss: 3.070023 Best loss: 0.111417 Accuracy: 95.97% 10 Validation loss: 5.498090 Best loss: 0.111417 Accuracy: 91.16% 11 Validation loss: 4.177420 Best loss: 0.111417 Accuracy: 91.99% 12 Validation loss: 2.052425 Best loss: 0.111417 Accuracy: 95.43% 13 Validation loss: 3.382188 Best loss: 0.111417 Accuracy: 92.89% 14 Validation loss: 2.243013 Best loss: 0.111417 Accuracy: 94.29% 15 Validation loss: 3.017085 Best loss: 0.111417 Accuracy: 94.37% 16 Validation loss: 1.125817 Best loss: 0.111417 Accuracy: 96.72% 17 Validation loss: 2.823200 Best loss: 0.111417 Accuracy: 94.68% 18 Validation loss: 2.504903 Best loss: 0.111417 Accuracy: 93.82% 19 Validation loss: 3.044997 Best loss: 0.111417 Accuracy: 95.31% 20 Validation loss: 1.800854 Best loss: 0.111417 Accuracy: 95.04% 21 Validation loss: 2.576041 Best loss: 0.111417 Accuracy: 96.91% 22 Validation loss: 2.695056 Best loss: 0.111417 Accuracy: 93.04% 23 Validation loss: 1.904910 Best loss: 0.111417 Accuracy: 94.61% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=100, learning_rate=0.05, total= 12.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02 0 Validation loss: 0.141793 Best loss: 0.141793 Accuracy: 95.86% 1 Validation loss: 0.143649 Best loss: 0.141793 Accuracy: 96.56% 2 Validation loss: 0.121541 Best loss: 0.121541 Accuracy: 96.72% 3 Validation loss: 0.195362 Best loss: 0.121541 Accuracy: 94.88% 4 Validation loss: 0.181279 Best loss: 0.121541 Accuracy: 95.19% 5 Validation loss: 0.170793 Best loss: 0.121541 Accuracy: 95.58% 6 Validation loss: 0.425543 Best loss: 0.121541 Accuracy: 79.79% 7 Validation loss: 0.392647 Best loss: 0.121541 Accuracy: 82.29% 8 Validation loss: 0.201778 Best loss: 0.121541 Accuracy: 95.39% 9 Validation loss: 0.163937 Best loss: 0.121541 Accuracy: 96.52% 10 Validation loss: 0.188088 Best loss: 0.121541 Accuracy: 95.78% 11 Validation loss: 0.408493 Best loss: 0.121541 Accuracy: 89.87% 12 Validation loss: 0.252903 Best loss: 0.121541 Accuracy: 95.11% 13 Validation loss: 0.268078 Best loss: 0.121541 Accuracy: 93.94% 14 Validation loss: 0.345477 Best loss: 0.121541 Accuracy: 91.05% 15 Validation loss: 0.300494 Best loss: 0.121541 Accuracy: 93.24% 16 Validation loss: 0.309207 Best loss: 0.121541 Accuracy: 93.71% 17 Validation loss: 0.507744 Best loss: 0.121541 Accuracy: 76.39% 18 Validation loss: 0.569249 Best loss: 0.121541 Accuracy: 74.43% 19 Validation loss: 0.444557 Best loss: 0.121541 Accuracy: 78.73% 20 Validation loss: 0.578371 Best loss: 0.121541 Accuracy: 75.68% 21 Validation loss: 0.520749 Best loss: 0.121541 Accuracy: 79.28% 22 Validation loss: 0.499509 Best loss: 0.121541 Accuracy: 76.62% 23 Validation loss: 0.456180 Best loss: 0.121541 Accuracy: 77.33% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02, total= 1.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02 0 Validation loss: 0.181112 Best loss: 0.181112 Accuracy: 94.02% 1 Validation loss: 0.147975 Best loss: 0.147975 Accuracy: 96.52% 2 Validation loss: 0.183324 Best loss: 0.147975 Accuracy: 95.90% 3 Validation loss: 0.157578 Best loss: 0.147975 Accuracy: 96.56% 4 Validation loss: 0.247682 Best loss: 0.147975 Accuracy: 94.76% 5 Validation loss: 0.158766 Best loss: 0.147975 Accuracy: 95.35% 6 Validation loss: 0.258877 Best loss: 0.147975 Accuracy: 94.10% 7 Validation loss: 0.163181 Best loss: 0.147975 Accuracy: 96.09% 8 Validation loss: 0.208652 Best loss: 0.147975 Accuracy: 95.74% 9 Validation loss: 0.187244 Best loss: 0.147975 Accuracy: 95.47% 10 Validation loss: 0.166194 Best loss: 0.147975 Accuracy: 96.09% 11 Validation loss: 0.251551 Best loss: 0.147975 Accuracy: 92.69% 12 Validation loss: 0.289218 Best loss: 0.147975 Accuracy: 94.88% 13 Validation loss: 0.219654 Best loss: 0.147975 Accuracy: 95.58% 14 Validation loss: 0.176042 Best loss: 0.147975 Accuracy: 96.33% 15 Validation loss: 0.463991 Best loss: 0.147975 Accuracy: 76.19% 16 Validation loss: 0.506136 Best loss: 0.147975 Accuracy: 75.92% 17 Validation loss: 0.852020 Best loss: 0.147975 Accuracy: 53.71% 18 Validation loss: 0.416372 Best loss: 0.147975 Accuracy: 76.47% 19 Validation loss: 0.422902 Best loss: 0.147975 Accuracy: 79.40% 20 Validation loss: 0.471068 Best loss: 0.147975 Accuracy: 74.82% 21 Validation loss: 0.247536 Best loss: 0.147975 Accuracy: 93.47% 22 Validation loss: 0.279639 Best loss: 0.147975 Accuracy: 94.49% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02, total= 1.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02 0 Validation loss: 0.267536 Best loss: 0.267536 Accuracy: 95.54% 1 Validation loss: 0.142089 Best loss: 0.142089 Accuracy: 96.87% 2 Validation loss: 0.129235 Best loss: 0.129235 Accuracy: 96.68% 3 Validation loss: 0.162777 Best loss: 0.129235 Accuracy: 95.54% 4 Validation loss: 0.180479 Best loss: 0.129235 Accuracy: 96.01% 5 Validation loss: 0.161733 Best loss: 0.129235 Accuracy: 96.05% 6 Validation loss: 0.148895 Best loss: 0.129235 Accuracy: 96.87% 7 Validation loss: 1.046287 Best loss: 0.129235 Accuracy: 49.80% 8 Validation loss: 0.722249 Best loss: 0.129235 Accuracy: 65.17% 9 Validation loss: 0.210944 Best loss: 0.129235 Accuracy: 94.88% 10 Validation loss: 0.189938 Best loss: 0.129235 Accuracy: 96.13% 11 Validation loss: 0.229616 Best loss: 0.129235 Accuracy: 96.25% 12 Validation loss: 0.209714 Best loss: 0.129235 Accuracy: 95.70% 13 Validation loss: 0.252285 Best loss: 0.129235 Accuracy: 95.04% 14 Validation loss: 2.724004 Best loss: 0.129235 Accuracy: 71.19% 15 Validation loss: 0.520483 Best loss: 0.129235 Accuracy: 77.76% 16 Validation loss: 0.494612 Best loss: 0.129235 Accuracy: 77.87% 17 Validation loss: 0.262035 Best loss: 0.129235 Accuracy: 95.00% 18 Validation loss: 0.273368 Best loss: 0.129235 Accuracy: 94.25% 19 Validation loss: 0.216133 Best loss: 0.129235 Accuracy: 95.47% 20 Validation loss: 0.358910 Best loss: 0.129235 Accuracy: 94.41% 21 Validation loss: 0.171805 Best loss: 0.129235 Accuracy: 96.72% 22 Validation loss: 0.237101 Best loss: 0.129235 Accuracy: 95.50% 23 Validation loss: 0.221922 Best loss: 0.129235 Accuracy: 95.90% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=10, learning_rate=0.02, total= 1.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05 0 Validation loss: 1.487467 Best loss: 1.487467 Accuracy: 89.56% 1 Validation loss: 0.333791 Best loss: 0.333791 Accuracy: 94.21% 2 Validation loss: 0.216949 Best loss: 0.216949 Accuracy: 94.45% 3 Validation loss: 4747.051758 Best loss: 0.216949 Accuracy: 21.81% 4 Validation loss: 7768.334473 Best loss: 0.216949 Accuracy: 34.99% 5 Validation loss: 207.870255 Best loss: 0.216949 Accuracy: 72.83% 6 Validation loss: 81.004395 Best loss: 0.216949 Accuracy: 74.24% 7 Validation loss: 39.101643 Best loss: 0.216949 Accuracy: 87.80% 8 Validation loss: 32.653088 Best loss: 0.216949 Accuracy: 88.58% 9 Validation loss: 19.054646 Best loss: 0.216949 Accuracy: 90.81% 10 Validation loss: 29.044310 Best loss: 0.216949 Accuracy: 84.25% 11 Validation loss: 18.254278 Best loss: 0.216949 Accuracy: 91.13% 12 Validation loss: 14.878347 Best loss: 0.216949 Accuracy: 91.32% 13 Validation loss: 17.447495 Best loss: 0.216949 Accuracy: 88.15% 14 Validation loss: 23.711472 Best loss: 0.216949 Accuracy: 87.49% 15 Validation loss: 16.486746 Best loss: 0.216949 Accuracy: 88.23% 16 Validation loss: 9.667856 Best loss: 0.216949 Accuracy: 93.98% 17 Validation loss: 25.992792 Best loss: 0.216949 Accuracy: 90.85% 18 Validation loss: 11.328872 Best loss: 0.216949 Accuracy: 94.57% 19 Validation loss: 11.417756 Best loss: 0.216949 Accuracy: 93.08% 20 Validation loss: 14.004346 Best loss: 0.216949 Accuracy: 92.26% 21 Validation loss: 9.161237 Best loss: 0.216949 Accuracy: 94.41% 22 Validation loss: 8.143062 Best loss: 0.216949 Accuracy: 94.80% 23 Validation loss: 8.332920 Best loss: 0.216949 Accuracy: 94.29% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05, total= 3.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05 0 Validation loss: 14.105901 Best loss: 14.105901 Accuracy: 56.18% 1 Validation loss: 1.377327 Best loss: 1.377327 Accuracy: 90.50% 2 Validation loss: 0.285004 Best loss: 0.285004 Accuracy: 95.04% 3 Validation loss: 0.184784 Best loss: 0.184784 Accuracy: 96.09% 4 Validation loss: 0.253436 Best loss: 0.184784 Accuracy: 94.14% 5 Validation loss: 0.170802 Best loss: 0.170802 Accuracy: 95.90% 6 Validation loss: 0.140308 Best loss: 0.140308 Accuracy: 96.64% 7 Validation loss: 0.125495 Best loss: 0.125495 Accuracy: 97.11% 8 Validation loss: 0.178656 Best loss: 0.125495 Accuracy: 95.47% 9 Validation loss: 0.113266 Best loss: 0.113266 Accuracy: 97.03% 10 Validation loss: 0.136996 Best loss: 0.113266 Accuracy: 96.60% 11 Validation loss: 0.103630 Best loss: 0.103630 Accuracy: 97.46% 12 Validation loss: 0.106114 Best loss: 0.103630 Accuracy: 97.46% 13 Validation loss: 0.107314 Best loss: 0.103630 Accuracy: 97.30% 14 Validation loss: 0.107960 Best loss: 0.103630 Accuracy: 97.07% 15 Validation loss: 0.091147 Best loss: 0.091147 Accuracy: 97.34% 16 Validation loss: 0.098727 Best loss: 0.091147 Accuracy: 97.58% 17 Validation loss: 0.108382 Best loss: 0.091147 Accuracy: 97.50% 18 Validation loss: 0.090315 Best loss: 0.090315 Accuracy: 98.01% 19 Validation loss: 0.088609 Best loss: 0.088609 Accuracy: 97.89% 20 Validation loss: 0.171356 Best loss: 0.088609 Accuracy: 97.58% 21 Validation loss: 0.100082 Best loss: 0.088609 Accuracy: 97.69% 22 Validation loss: 0.092228 Best loss: 0.088609 Accuracy: 97.46% 23 Validation loss: 0.087911 Best loss: 0.087911 Accuracy: 97.65% 24 Validation loss: 0.091225 Best loss: 0.087911 Accuracy: 97.54% 25 Validation loss: 0.074923 Best loss: 0.074923 Accuracy: 98.08% 26 Validation loss: 0.075225 Best loss: 0.074923 Accuracy: 98.20% 27 Validation loss: 0.085478 Best loss: 0.074923 Accuracy: 98.05% 28 Validation loss: 0.079967 Best loss: 0.074923 Accuracy: 98.16% 29 Validation loss: 0.088529 Best loss: 0.074923 Accuracy: 98.08% 30 Validation loss: 0.211331 Best loss: 0.074923 Accuracy: 97.38% 31 Validation loss: 0.079630 Best loss: 0.074923 Accuracy: 98.20% 32 Validation loss: 326841.843750 Best loss: 0.074923 Accuracy: 18.73% 33 Validation loss: 4640431.500000 Best loss: 0.074923 Accuracy: 26.90% 34 Validation loss: 30926.486328 Best loss: 0.074923 Accuracy: 82.17% 35 Validation loss: 5446.049316 Best loss: 0.074923 Accuracy: 88.86% 36 Validation loss: 2814.811768 Best loss: 0.074923 Accuracy: 91.91% 37 Validation loss: 2647.902344 Best loss: 0.074923 Accuracy: 89.76% 38 Validation loss: 1890.773682 Best loss: 0.074923 Accuracy: 91.48% 39 Validation loss: 2488.987793 Best loss: 0.074923 Accuracy: 87.84% 40 Validation loss: 1505.058594 Best loss: 0.074923 Accuracy: 92.77% 41 Validation loss: 1361.767700 Best loss: 0.074923 Accuracy: 92.81% 42 Validation loss: 2023.896606 Best loss: 0.074923 Accuracy: 91.20% 43 Validation loss: 1259.026123 Best loss: 0.074923 Accuracy: 93.00% 44 Validation loss: 1180.648315 Best loss: 0.074923 Accuracy: 92.46% 45 Validation loss: 1234.874878 Best loss: 0.074923 Accuracy: 90.66% 46 Validation loss: 1216.927490 Best loss: 0.074923 Accuracy: 90.34% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05, total= 7.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05 0 Validation loss: 93.126457 Best loss: 93.126457 Accuracy: 29.98% 1 Validation loss: 14.775363 Best loss: 14.775363 Accuracy: 75.41% 2 Validation loss: 1.423933 Best loss: 1.423933 Accuracy: 90.07% 3 Validation loss: 0.634541 Best loss: 0.634541 Accuracy: 89.95% 4 Validation loss: 0.259469 Best loss: 0.259469 Accuracy: 96.05% 5 Validation loss: 0.245238 Best loss: 0.245238 Accuracy: 95.70% 6 Validation loss: 0.203192 Best loss: 0.203192 Accuracy: 96.17% 7 Validation loss: 0.209211 Best loss: 0.203192 Accuracy: 96.44% 8 Validation loss: 0.257455 Best loss: 0.203192 Accuracy: 95.78% 9 Validation loss: 0.220129 Best loss: 0.203192 Accuracy: 96.40% 10 Validation loss: 0.178897 Best loss: 0.178897 Accuracy: 96.99% 11 Validation loss: 0.211863 Best loss: 0.178897 Accuracy: 96.99% 12 Validation loss: 0.173348 Best loss: 0.173348 Accuracy: 97.58% 13 Validation loss: 0.202587 Best loss: 0.173348 Accuracy: 97.89% 14 Validation loss: 0.243739 Best loss: 0.173348 Accuracy: 97.81% 15 Validation loss: 0.274636 Best loss: 0.173348 Accuracy: 97.11% 16 Validation loss: 0.245023 Best loss: 0.173348 Accuracy: 97.07% 17 Validation loss: 0.204245 Best loss: 0.173348 Accuracy: 97.93% 18 Validation loss: 0.285605 Best loss: 0.173348 Accuracy: 97.22% 19 Validation loss: 0.180616 Best loss: 0.173348 Accuracy: 97.97% 20 Validation loss: 0.121456 Best loss: 0.121456 Accuracy: 98.28% 21 Validation loss: 0.148668 Best loss: 0.121456 Accuracy: 97.85% 22 Validation loss: 0.155988 Best loss: 0.121456 Accuracy: 98.01% 23 Validation loss: 0.126629 Best loss: 0.121456 Accuracy: 98.20% 24 Validation loss: 0.151432 Best loss: 0.121456 Accuracy: 98.28% 25 Validation loss: 0.139392 Best loss: 0.121456 Accuracy: 98.32% 26 Validation loss: 0.143437 Best loss: 0.121456 Accuracy: 98.16% 27 Validation loss: 0.148519 Best loss: 0.121456 Accuracy: 98.20% 28 Validation loss: 0.407344 Best loss: 0.121456 Accuracy: 98.55% 29 Validation loss: 0.541370 Best loss: 0.121456 Accuracy: 98.51% 30 Validation loss: 0.697161 Best loss: 0.121456 Accuracy: 98.32% 31 Validation loss: 0.322625 Best loss: 0.121456 Accuracy: 98.40% 32 Validation loss: 0.625466 Best loss: 0.121456 Accuracy: 98.16% 33 Validation loss: 0.535191 Best loss: 0.121456 Accuracy: 98.40% 34 Validation loss: 0.631373 Best loss: 0.121456 Accuracy: 98.01% 35 Validation loss: 0.707316 Best loss: 0.121456 Accuracy: 98.16% 36 Validation loss: 0.376325 Best loss: 0.121456 Accuracy: 97.54% 37 Validation loss: 0.454737 Best loss: 0.121456 Accuracy: 95.47% 38 Validation loss: 0.189824 Best loss: 0.121456 Accuracy: 97.69% 39 Validation loss: 0.157454 Best loss: 0.121456 Accuracy: 98.36% 40 Validation loss: 0.142956 Best loss: 0.121456 Accuracy: 98.16% 41 Validation loss: 0.131146 Best loss: 0.121456 Accuracy: 98.28% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.05, total= 6.0s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05 0 Validation loss: 0.272541 Best loss: 0.272541 Accuracy: 93.82% 1 Validation loss: 0.334694 Best loss: 0.272541 Accuracy: 91.95% 2 Validation loss: 0.215927 Best loss: 0.215927 Accuracy: 95.19% 3 Validation loss: 0.231539 Best loss: 0.215927 Accuracy: 92.69% 4 Validation loss: 0.182777 Best loss: 0.182777 Accuracy: 95.93% 5 Validation loss: 0.197816 Best loss: 0.182777 Accuracy: 94.72% 6 Validation loss: 0.260721 Best loss: 0.182777 Accuracy: 93.71% 7 Validation loss: 0.248328 Best loss: 0.182777 Accuracy: 93.90% 8 Validation loss: 0.206701 Best loss: 0.182777 Accuracy: 95.43% 9 Validation loss: 0.245235 Best loss: 0.182777 Accuracy: 95.15% 10 Validation loss: 0.218311 Best loss: 0.182777 Accuracy: 96.09% 11 Validation loss: 0.249565 Best loss: 0.182777 Accuracy: 94.25% 12 Validation loss: 0.217278 Best loss: 0.182777 Accuracy: 94.92% 13 Validation loss: 0.261426 Best loss: 0.182777 Accuracy: 95.39% 14 Validation loss: 0.250647 Best loss: 0.182777 Accuracy: 95.15% 15 Validation loss: 0.265926 Best loss: 0.182777 Accuracy: 93.24% 16 Validation loss: 0.366209 Best loss: 0.182777 Accuracy: 93.16% 17 Validation loss: 0.267726 Best loss: 0.182777 Accuracy: 94.45% 18 Validation loss: 0.236130 Best loss: 0.182777 Accuracy: 94.96% 19 Validation loss: 0.271614 Best loss: 0.182777 Accuracy: 94.25% 20 Validation loss: 0.246886 Best loss: 0.182777 Accuracy: 95.35% 21 Validation loss: 168.615295 Best loss: 0.182777 Accuracy: 74.78% 22 Validation loss: 1.254972 Best loss: 0.182777 Accuracy: 38.39% 23 Validation loss: 1.311199 Best loss: 0.182777 Accuracy: 40.27% 24 Validation loss: 1.137057 Best loss: 0.182777 Accuracy: 42.22% 25 Validation loss: 1.132979 Best loss: 0.182777 Accuracy: 40.03% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 10.9s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05 0 Validation loss: 0.710941 Best loss: 0.710941 Accuracy: 77.72% 1 Validation loss: 0.437987 Best loss: 0.437987 Accuracy: 91.24% 2 Validation loss: 0.403740 Best loss: 0.403740 Accuracy: 91.87% 3 Validation loss: 0.378467 Best loss: 0.378467 Accuracy: 93.94% 4 Validation loss: 0.350204 Best loss: 0.350204 Accuracy: 94.72% 5 Validation loss: 0.590498 Best loss: 0.350204 Accuracy: 80.69% 6 Validation loss: 0.447232 Best loss: 0.350204 Accuracy: 95.70% 7 Validation loss: 0.338170 Best loss: 0.338170 Accuracy: 94.25% 8 Validation loss: 0.329199 Best loss: 0.329199 Accuracy: 94.72% 9 Validation loss: 0.422246 Best loss: 0.329199 Accuracy: 94.29% 10 Validation loss: 0.414066 Best loss: 0.329199 Accuracy: 93.16% 11 Validation loss: 0.422891 Best loss: 0.329199 Accuracy: 95.78% 12 Validation loss: 0.383372 Best loss: 0.329199 Accuracy: 95.04% 13 Validation loss: 0.457702 Best loss: 0.329199 Accuracy: 95.43% 14 Validation loss: 0.426164 Best loss: 0.329199 Accuracy: 95.54% 15 Validation loss: 0.370727 Best loss: 0.329199 Accuracy: 94.72% 16 Validation loss: 0.497654 Best loss: 0.329199 Accuracy: 95.23% 17 Validation loss: 0.456721 Best loss: 0.329199 Accuracy: 95.47% 18 Validation loss: 0.350218 Best loss: 0.329199 Accuracy: 95.58% 19 Validation loss: 0.787020 Best loss: 0.329199 Accuracy: 95.47% 20 Validation loss: 0.516814 Best loss: 0.329199 Accuracy: 95.90% 21 Validation loss: 0.405833 Best loss: 0.329199 Accuracy: 96.25% 22 Validation loss: 0.567135 Best loss: 0.329199 Accuracy: 95.78% 23 Validation loss: 0.440018 Best loss: 0.329199 Accuracy: 95.50% 24 Validation loss: 0.563025 Best loss: 0.329199 Accuracy: 93.24% 25 Validation loss: 0.486234 Best loss: 0.329199 Accuracy: 95.82% 26 Validation loss: 0.476396 Best loss: 0.329199 Accuracy: 95.97% 27 Validation loss: 0.617170 Best loss: 0.329199 Accuracy: 96.25% 28 Validation loss: 0.386440 Best loss: 0.329199 Accuracy: 93.90% 29 Validation loss: 0.398929 Best loss: 0.329199 Accuracy: 96.36% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 12.5s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05 0 Validation loss: 0.153212 Best loss: 0.153212 Accuracy: 96.36% 1 Validation loss: 0.199441 Best loss: 0.153212 Accuracy: 95.90% 2 Validation loss: 0.288099 Best loss: 0.153212 Accuracy: 94.45% 3 Validation loss: 0.172885 Best loss: 0.153212 Accuracy: 95.27% 4 Validation loss: 0.159583 Best loss: 0.153212 Accuracy: 95.27% 5 Validation loss: 0.485513 Best loss: 0.153212 Accuracy: 88.19% 6 Validation loss: 0.248562 Best loss: 0.153212 Accuracy: 93.86% 7 Validation loss: 0.482406 Best loss: 0.153212 Accuracy: 82.41% 8 Validation loss: 0.818394 Best loss: 0.153212 Accuracy: 59.85% 9 Validation loss: 1.430300 Best loss: 0.153212 Accuracy: 39.01% 10 Validation loss: 1.176267 Best loss: 0.153212 Accuracy: 40.54% 11 Validation loss: 1.168755 Best loss: 0.153212 Accuracy: 40.23% 12 Validation loss: 1.196880 Best loss: 0.153212 Accuracy: 40.19% 13 Validation loss: 1.184643 Best loss: 0.153212 Accuracy: 37.37% 14 Validation loss: 1.252619 Best loss: 0.153212 Accuracy: 38.04% 15 Validation loss: 1.183608 Best loss: 0.153212 Accuracy: 40.19% 16 Validation loss: 0.850975 Best loss: 0.153212 Accuracy: 63.14% 17 Validation loss: 0.805377 Best loss: 0.153212 Accuracy: 67.83% 18 Validation loss: 0.632088 Best loss: 0.153212 Accuracy: 74.43% 19 Validation loss: 0.727640 Best loss: 0.153212 Accuracy: 68.92% 20 Validation loss: 0.740954 Best loss: 0.153212 Accuracy: 65.75% 21 Validation loss: 0.652964 Best loss: 0.153212 Accuracy: 74.32% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.05, total= 9.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02 0 Validation loss: 0.284828 Best loss: 0.284828 Accuracy: 90.62% 1 Validation loss: 0.138043 Best loss: 0.138043 Accuracy: 95.97% 2 Validation loss: 0.107018 Best loss: 0.107018 Accuracy: 96.91% 3 Validation loss: 0.110573 Best loss: 0.107018 Accuracy: 96.72% 4 Validation loss: 0.095894 Best loss: 0.095894 Accuracy: 97.19% 5 Validation loss: 0.079891 Best loss: 0.079891 Accuracy: 97.58% 6 Validation loss: 0.080245 Best loss: 0.079891 Accuracy: 97.65% 7 Validation loss: 0.086206 Best loss: 0.079891 Accuracy: 97.77% 8 Validation loss: 0.128917 Best loss: 0.079891 Accuracy: 96.40% 9 Validation loss: 0.086298 Best loss: 0.079891 Accuracy: 97.54% 10 Validation loss: 0.078643 Best loss: 0.078643 Accuracy: 97.81% 11 Validation loss: 0.079902 Best loss: 0.078643 Accuracy: 97.69% 12 Validation loss: 0.079222 Best loss: 0.078643 Accuracy: 97.93% 13 Validation loss: 0.089221 Best loss: 0.078643 Accuracy: 97.81% 14 Validation loss: 0.119608 Best loss: 0.078643 Accuracy: 97.19% 15 Validation loss: 0.091027 Best loss: 0.078643 Accuracy: 98.01% 16 Validation loss: 0.090057 Best loss: 0.078643 Accuracy: 98.12% 17 Validation loss: 0.100548 Best loss: 0.078643 Accuracy: 97.97% 18 Validation loss: 0.099989 Best loss: 0.078643 Accuracy: 97.73% 19 Validation loss: 0.085788 Best loss: 0.078643 Accuracy: 98.01% 20 Validation loss: 0.094321 Best loss: 0.078643 Accuracy: 98.08% 21 Validation loss: 0.098117 Best loss: 0.078643 Accuracy: 97.97% 22 Validation loss: 0.099268 Best loss: 0.078643 Accuracy: 98.01% 23 Validation loss: 0.094390 Best loss: 0.078643 Accuracy: 98.28% 24 Validation loss: 0.119017 Best loss: 0.078643 Accuracy: 97.50% 25 Validation loss: 0.085257 Best loss: 0.078643 Accuracy: 98.01% 26 Validation loss: 0.105676 Best loss: 0.078643 Accuracy: 97.54% 27 Validation loss: 0.083218 Best loss: 0.078643 Accuracy: 98.24% 28 Validation loss: 0.099552 Best loss: 0.078643 Accuracy: 97.97% 29 Validation loss: 0.113790 Best loss: 0.078643 Accuracy: 97.93% 30 Validation loss: 0.098753 Best loss: 0.078643 Accuracy: 98.28% 31 Validation loss: 0.099818 Best loss: 0.078643 Accuracy: 98.24% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02, total= 4.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02 0 Validation loss: 0.240080 Best loss: 0.240080 Accuracy: 93.35% 1 Validation loss: 0.114037 Best loss: 0.114037 Accuracy: 96.56% 2 Validation loss: 0.087474 Best loss: 0.087474 Accuracy: 97.50% 3 Validation loss: 0.083461 Best loss: 0.083461 Accuracy: 97.42% 4 Validation loss: 0.078342 Best loss: 0.078342 Accuracy: 97.73% 5 Validation loss: 0.093510 Best loss: 0.078342 Accuracy: 97.03% 6 Validation loss: 0.084399 Best loss: 0.078342 Accuracy: 97.46% 7 Validation loss: 0.087539 Best loss: 0.078342 Accuracy: 97.22% 8 Validation loss: 0.091221 Best loss: 0.078342 Accuracy: 97.42% 9 Validation loss: 0.079019 Best loss: 0.078342 Accuracy: 97.93% 10 Validation loss: 0.080337 Best loss: 0.078342 Accuracy: 98.05% 11 Validation loss: 0.077136 Best loss: 0.077136 Accuracy: 97.93% 12 Validation loss: 0.079500 Best loss: 0.077136 Accuracy: 98.05% 13 Validation loss: 0.132179 Best loss: 0.077136 Accuracy: 96.95% 14 Validation loss: 0.078672 Best loss: 0.077136 Accuracy: 97.93% 15 Validation loss: 0.086034 Best loss: 0.077136 Accuracy: 98.08% 16 Validation loss: 0.098404 Best loss: 0.077136 Accuracy: 97.85% 17 Validation loss: 0.087914 Best loss: 0.077136 Accuracy: 97.85% 18 Validation loss: 0.087968 Best loss: 0.077136 Accuracy: 97.81% 19 Validation loss: 0.114619 Best loss: 0.077136 Accuracy: 97.65% 20 Validation loss: 0.096992 Best loss: 0.077136 Accuracy: 98.01% 21 Validation loss: 0.081783 Best loss: 0.077136 Accuracy: 98.01% 22 Validation loss: 0.089468 Best loss: 0.077136 Accuracy: 98.05% 23 Validation loss: 0.112918 Best loss: 0.077136 Accuracy: 97.54% 24 Validation loss: 0.093693 Best loss: 0.077136 Accuracy: 98.20% 25 Validation loss: 0.097276 Best loss: 0.077136 Accuracy: 98.01% 26 Validation loss: 0.104344 Best loss: 0.077136 Accuracy: 97.89% 27 Validation loss: 0.100905 Best loss: 0.077136 Accuracy: 97.89% 28 Validation loss: 0.111584 Best loss: 0.077136 Accuracy: 97.77% 29 Validation loss: 0.135019 Best loss: 0.077136 Accuracy: 97.54% 30 Validation loss: 0.092633 Best loss: 0.077136 Accuracy: 98.05% 31 Validation loss: 0.111303 Best loss: 0.077136 Accuracy: 97.93% 32 Validation loss: 0.121083 Best loss: 0.077136 Accuracy: 98.05% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02, total= 5.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02 0 Validation loss: 0.299604 Best loss: 0.299604 Accuracy: 90.97% 1 Validation loss: 0.132859 Best loss: 0.132859 Accuracy: 96.17% 2 Validation loss: 0.100547 Best loss: 0.100547 Accuracy: 97.15% 3 Validation loss: 0.095562 Best loss: 0.095562 Accuracy: 97.22% 4 Validation loss: 0.110037 Best loss: 0.095562 Accuracy: 97.22% 5 Validation loss: 0.075004 Best loss: 0.075004 Accuracy: 97.97% 6 Validation loss: 0.087895 Best loss: 0.075004 Accuracy: 97.81% 7 Validation loss: 0.086601 Best loss: 0.075004 Accuracy: 97.34% 8 Validation loss: 0.073188 Best loss: 0.073188 Accuracy: 97.97% 9 Validation loss: 0.070803 Best loss: 0.070803 Accuracy: 98.16% 10 Validation loss: 0.064092 Best loss: 0.064092 Accuracy: 98.55% 11 Validation loss: 0.073642 Best loss: 0.064092 Accuracy: 98.01% 12 Validation loss: 0.068739 Best loss: 0.064092 Accuracy: 98.05% 13 Validation loss: 0.070418 Best loss: 0.064092 Accuracy: 98.05% 14 Validation loss: 0.074784 Best loss: 0.064092 Accuracy: 98.20% 15 Validation loss: 0.067433 Best loss: 0.064092 Accuracy: 98.20% 16 Validation loss: 0.075162 Best loss: 0.064092 Accuracy: 98.05% 17 Validation loss: 0.076716 Best loss: 0.064092 Accuracy: 98.16% 18 Validation loss: 0.065946 Best loss: 0.064092 Accuracy: 98.32% 19 Validation loss: 0.058545 Best loss: 0.058545 Accuracy: 98.63% 20 Validation loss: 0.070492 Best loss: 0.058545 Accuracy: 98.32% 21 Validation loss: 0.072676 Best loss: 0.058545 Accuracy: 98.36% 22 Validation loss: 0.070992 Best loss: 0.058545 Accuracy: 98.40% 23 Validation loss: 0.072800 Best loss: 0.058545 Accuracy: 98.36% 24 Validation loss: 0.078711 Best loss: 0.058545 Accuracy: 98.05% 25 Validation loss: 0.083881 Best loss: 0.058545 Accuracy: 97.77% 26 Validation loss: 0.082218 Best loss: 0.058545 Accuracy: 98.20% 27 Validation loss: 0.070453 Best loss: 0.058545 Accuracy: 98.20% 28 Validation loss: 0.073779 Best loss: 0.058545 Accuracy: 98.12% 29 Validation loss: 0.074992 Best loss: 0.058545 Accuracy: 98.24% 30 Validation loss: 0.076958 Best loss: 0.058545 Accuracy: 98.12% 31 Validation loss: 0.077275 Best loss: 0.058545 Accuracy: 98.28% 32 Validation loss: 0.072716 Best loss: 0.058545 Accuracy: 98.08% 33 Validation loss: 0.080357 Best loss: 0.058545 Accuracy: 97.89% 34 Validation loss: 0.088416 Best loss: 0.058545 Accuracy: 98.01% 35 Validation loss: 0.071452 Best loss: 0.058545 Accuracy: 98.36% 36 Validation loss: 0.083622 Best loss: 0.058545 Accuracy: 98.16% 37 Validation loss: 0.084725 Best loss: 0.058545 Accuracy: 98.20% 38 Validation loss: 0.091014 Best loss: 0.058545 Accuracy: 98.01% 39 Validation loss: 0.100796 Best loss: 0.058545 Accuracy: 97.73% 40 Validation loss: 0.083031 Best loss: 0.058545 Accuracy: 98.28% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=10, batch_size=500, learning_rate=0.02, total= 6.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.119916 Best loss: 0.119916 Accuracy: 96.09% 1 Validation loss: 0.100660 Best loss: 0.100660 Accuracy: 97.30% 2 Validation loss: 0.071796 Best loss: 0.071796 Accuracy: 97.97% 3 Validation loss: 1.424624 Best loss: 0.071796 Accuracy: 77.56% 4 Validation loss: 1.699469 Best loss: 0.071796 Accuracy: 19.27% 5 Validation loss: 1.640345 Best loss: 0.071796 Accuracy: 18.73% 6 Validation loss: 1.660829 Best loss: 0.071796 Accuracy: 19.27% 7 Validation loss: 1.632374 Best loss: 0.071796 Accuracy: 22.01% 8 Validation loss: 1.639604 Best loss: 0.071796 Accuracy: 19.27% 9 Validation loss: 1.661292 Best loss: 0.071796 Accuracy: 22.01% 10 Validation loss: 1.630342 Best loss: 0.071796 Accuracy: 20.91% 11 Validation loss: 1.626308 Best loss: 0.071796 Accuracy: 22.01% 12 Validation loss: 1.636569 Best loss: 0.071796 Accuracy: 20.91% 13 Validation loss: 1.616861 Best loss: 0.071796 Accuracy: 20.91% 14 Validation loss: 1.624239 Best loss: 0.071796 Accuracy: 19.08% 15 Validation loss: 1.619350 Best loss: 0.071796 Accuracy: 22.01% 16 Validation loss: 1.631293 Best loss: 0.071796 Accuracy: 18.73% 17 Validation loss: 1.628955 Best loss: 0.071796 Accuracy: 19.27% 18 Validation loss: 1.681627 Best loss: 0.071796 Accuracy: 22.01% 19 Validation loss: 1.618935 Best loss: 0.071796 Accuracy: 22.01% 20 Validation loss: 1.640259 Best loss: 0.071796 Accuracy: 19.27% 21 Validation loss: 1.657086 Best loss: 0.071796 Accuracy: 19.08% 22 Validation loss: 1.636957 Best loss: 0.071796 Accuracy: 22.01% 23 Validation loss: 1.619310 Best loss: 0.071796 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 10.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.090135 Best loss: 0.090135 Accuracy: 97.42% 1 Validation loss: 0.099018 Best loss: 0.090135 Accuracy: 97.22% 2 Validation loss: 0.093783 Best loss: 0.090135 Accuracy: 98.12% 3 Validation loss: 0.220947 Best loss: 0.090135 Accuracy: 93.67% 4 Validation loss: 1.664270 Best loss: 0.090135 Accuracy: 19.08% 5 Validation loss: 1.741650 Best loss: 0.090135 Accuracy: 22.01% 6 Validation loss: 1.653350 Best loss: 0.090135 Accuracy: 19.27% 7 Validation loss: 1.628280 Best loss: 0.090135 Accuracy: 18.73% 8 Validation loss: 1.634682 Best loss: 0.090135 Accuracy: 19.27% 9 Validation loss: 1.672642 Best loss: 0.090135 Accuracy: 19.27% 10 Validation loss: 1.615055 Best loss: 0.090135 Accuracy: 19.27% 11 Validation loss: 1.647010 Best loss: 0.090135 Accuracy: 19.27% 12 Validation loss: 1.622361 Best loss: 0.090135 Accuracy: 22.01% 13 Validation loss: 1.633268 Best loss: 0.090135 Accuracy: 18.73% 14 Validation loss: 1.629335 Best loss: 0.090135 Accuracy: 22.01% 15 Validation loss: 1.619790 Best loss: 0.090135 Accuracy: 19.08% 16 Validation loss: 1.625546 Best loss: 0.090135 Accuracy: 19.27% 17 Validation loss: 1.644742 Best loss: 0.090135 Accuracy: 20.91% 18 Validation loss: 1.672329 Best loss: 0.090135 Accuracy: 22.01% 19 Validation loss: 1.621137 Best loss: 0.090135 Accuracy: 19.27% 20 Validation loss: 1.629635 Best loss: 0.090135 Accuracy: 18.73% 21 Validation loss: 1.712916 Best loss: 0.090135 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 9.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.135077 Best loss: 0.135077 Accuracy: 96.48% 1 Validation loss: 0.135246 Best loss: 0.135077 Accuracy: 96.72% 2 Validation loss: 0.083968 Best loss: 0.083968 Accuracy: 98.01% 3 Validation loss: 0.106711 Best loss: 0.083968 Accuracy: 97.85% 4 Validation loss: 1.639282 Best loss: 0.083968 Accuracy: 22.01% 5 Validation loss: 1.689417 Best loss: 0.083968 Accuracy: 22.01% 6 Validation loss: 1.615560 Best loss: 0.083968 Accuracy: 19.08% 7 Validation loss: 1.648099 Best loss: 0.083968 Accuracy: 22.01% 8 Validation loss: 1.652037 Best loss: 0.083968 Accuracy: 19.27% 9 Validation loss: 1.675255 Best loss: 0.083968 Accuracy: 22.01% 10 Validation loss: 1.652091 Best loss: 0.083968 Accuracy: 22.01% 11 Validation loss: 1.652307 Best loss: 0.083968 Accuracy: 19.27% 12 Validation loss: 1.643352 Best loss: 0.083968 Accuracy: 20.91% 13 Validation loss: 1.623650 Best loss: 0.083968 Accuracy: 19.08% 14 Validation loss: 1.645207 Best loss: 0.083968 Accuracy: 19.08% 15 Validation loss: 1.661911 Best loss: 0.083968 Accuracy: 20.91% 16 Validation loss: 1.691771 Best loss: 0.083968 Accuracy: 18.73% 17 Validation loss: 1.659158 Best loss: 0.083968 Accuracy: 20.91% 18 Validation loss: 1.663521 Best loss: 0.083968 Accuracy: 18.73% 19 Validation loss: 1.616484 Best loss: 0.083968 Accuracy: 20.91% 20 Validation loss: 1.611590 Best loss: 0.083968 Accuracy: 19.27% 21 Validation loss: 1.695784 Best loss: 0.083968 Accuracy: 22.01% 22 Validation loss: 1.621100 Best loss: 0.083968 Accuracy: 19.08% 23 Validation loss: 1.710887 Best loss: 0.083968 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 10.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.099367 Best loss: 0.099367 Accuracy: 96.76% 1 Validation loss: 0.103293 Best loss: 0.099367 Accuracy: 97.38% 2 Validation loss: 0.074610 Best loss: 0.074610 Accuracy: 97.85% 3 Validation loss: 0.083603 Best loss: 0.074610 Accuracy: 97.54% 4 Validation loss: 0.082414 Best loss: 0.074610 Accuracy: 97.93% 5 Validation loss: 67.566391 Best loss: 0.074610 Accuracy: 74.20% 6 Validation loss: 0.129333 Best loss: 0.074610 Accuracy: 96.40% 7 Validation loss: 0.115857 Best loss: 0.074610 Accuracy: 96.36% 8 Validation loss: 0.089923 Best loss: 0.074610 Accuracy: 97.69% 9 Validation loss: 0.086107 Best loss: 0.074610 Accuracy: 97.81% 10 Validation loss: 0.086021 Best loss: 0.074610 Accuracy: 98.01% 11 Validation loss: 0.085553 Best loss: 0.074610 Accuracy: 97.73% 12 Validation loss: 0.083494 Best loss: 0.074610 Accuracy: 97.97% 13 Validation loss: 0.086281 Best loss: 0.074610 Accuracy: 98.08% 14 Validation loss: 0.095160 Best loss: 0.074610 Accuracy: 97.97% 15 Validation loss: 0.085932 Best loss: 0.074610 Accuracy: 98.24% 16 Validation loss: 0.090309 Best loss: 0.074610 Accuracy: 98.12% 17 Validation loss: 0.072857 Best loss: 0.072857 Accuracy: 98.40% 18 Validation loss: 0.109465 Best loss: 0.072857 Accuracy: 97.81% 19 Validation loss: 0.086737 Best loss: 0.072857 Accuracy: 98.28% 20 Validation loss: 0.142684 Best loss: 0.072857 Accuracy: 98.24% 21 Validation loss: 0.116292 Best loss: 0.072857 Accuracy: 98.20% 22 Validation loss: 0.084377 Best loss: 0.072857 Accuracy: 98.44% 23 Validation loss: 0.077182 Best loss: 0.072857 Accuracy: 98.55% 24 Validation loss: 0.089147 Best loss: 0.072857 Accuracy: 98.16% 25 Validation loss: 0.074125 Best loss: 0.072857 Accuracy: 98.40% 26 Validation loss: 0.094374 Best loss: 0.072857 Accuracy: 98.51% 27 Validation loss: 0.119456 Best loss: 0.072857 Accuracy: 98.44% 28 Validation loss: 0.122412 Best loss: 0.072857 Accuracy: 97.73% 29 Validation loss: 3.362872 Best loss: 0.072857 Accuracy: 93.47% 30 Validation loss: 0.172960 Best loss: 0.072857 Accuracy: 96.17% 31 Validation loss: 0.291616 Best loss: 0.072857 Accuracy: 95.15% 32 Validation loss: 0.104338 Best loss: 0.072857 Accuracy: 97.07% 33 Validation loss: 0.097650 Best loss: 0.072857 Accuracy: 97.38% 34 Validation loss: 0.094906 Best loss: 0.072857 Accuracy: 97.58% 35 Validation loss: 0.093949 Best loss: 0.072857 Accuracy: 97.42% 36 Validation loss: 0.091386 Best loss: 0.072857 Accuracy: 97.81% 37 Validation loss: 0.093877 Best loss: 0.072857 Accuracy: 97.81% 38 Validation loss: 0.092394 Best loss: 0.072857 Accuracy: 97.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 19.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.107817 Best loss: 0.107817 Accuracy: 96.83% 1 Validation loss: 0.070397 Best loss: 0.070397 Accuracy: 97.97% 2 Validation loss: 0.099165 Best loss: 0.070397 Accuracy: 97.54% 3 Validation loss: 0.065494 Best loss: 0.065494 Accuracy: 98.28% 4 Validation loss: 0.084656 Best loss: 0.065494 Accuracy: 97.97% 5 Validation loss: 0.077864 Best loss: 0.065494 Accuracy: 97.93% 6 Validation loss: 0.221358 Best loss: 0.065494 Accuracy: 91.16% 7 Validation loss: 0.319363 Best loss: 0.065494 Accuracy: 89.80% 8 Validation loss: 0.528356 Best loss: 0.065494 Accuracy: 93.78% 9 Validation loss: 0.115311 Best loss: 0.065494 Accuracy: 96.95% 10 Validation loss: 0.095289 Best loss: 0.065494 Accuracy: 97.50% 11 Validation loss: 0.088154 Best loss: 0.065494 Accuracy: 97.46% 12 Validation loss: 0.080160 Best loss: 0.065494 Accuracy: 97.73% 13 Validation loss: 0.077240 Best loss: 0.065494 Accuracy: 98.01% 14 Validation loss: 0.100681 Best loss: 0.065494 Accuracy: 97.69% 15 Validation loss: 0.071279 Best loss: 0.065494 Accuracy: 98.08% 16 Validation loss: 0.098273 Best loss: 0.065494 Accuracy: 97.58% 17 Validation loss: 0.112732 Best loss: 0.065494 Accuracy: 97.65% 18 Validation loss: 0.101835 Best loss: 0.065494 Accuracy: 97.07% 19 Validation loss: 0.088993 Best loss: 0.065494 Accuracy: 97.89% 20 Validation loss: 0.083926 Best loss: 0.065494 Accuracy: 97.73% 21 Validation loss: 0.088431 Best loss: 0.065494 Accuracy: 98.12% 22 Validation loss: 0.095673 Best loss: 0.065494 Accuracy: 98.28% 23 Validation loss: 0.098625 Best loss: 0.065494 Accuracy: 97.77% 24 Validation loss: 0.117960 Best loss: 0.065494 Accuracy: 97.42% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 13.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02 0 Validation loss: 0.106338 Best loss: 0.106338 Accuracy: 97.15% 1 Validation loss: 0.109764 Best loss: 0.106338 Accuracy: 97.15% 2 Validation loss: 0.076569 Best loss: 0.076569 Accuracy: 97.81% 3 Validation loss: 0.082875 Best loss: 0.076569 Accuracy: 98.01% 4 Validation loss: 0.759423 Best loss: 0.076569 Accuracy: 89.76% 5 Validation loss: 0.138289 Best loss: 0.076569 Accuracy: 96.33% 6 Validation loss: 0.110148 Best loss: 0.076569 Accuracy: 97.11% 7 Validation loss: 0.084210 Best loss: 0.076569 Accuracy: 97.58% 8 Validation loss: 0.080925 Best loss: 0.076569 Accuracy: 97.77% 9 Validation loss: 0.261730 Best loss: 0.076569 Accuracy: 94.84% 10 Validation loss: 0.095278 Best loss: 0.076569 Accuracy: 97.54% 11 Validation loss: 0.090194 Best loss: 0.076569 Accuracy: 97.42% 12 Validation loss: 0.108749 Best loss: 0.076569 Accuracy: 97.81% 13 Validation loss: 0.142064 Best loss: 0.076569 Accuracy: 96.91% 14 Validation loss: 0.084360 Best loss: 0.076569 Accuracy: 97.89% 15 Validation loss: 0.149550 Best loss: 0.076569 Accuracy: 98.08% 16 Validation loss: 0.119191 Best loss: 0.076569 Accuracy: 97.93% 17 Validation loss: 0.708432 Best loss: 0.076569 Accuracy: 98.12% 18 Validation loss: 0.141218 Best loss: 0.076569 Accuracy: 96.76% 19 Validation loss: 10.140091 Best loss: 0.076569 Accuracy: 87.57% 20 Validation loss: 0.413241 Best loss: 0.076569 Accuracy: 94.92% 21 Validation loss: 0.300595 Best loss: 0.076569 Accuracy: 95.82% 22 Validation loss: 0.310090 Best loss: 0.076569 Accuracy: 94.64% 23 Validation loss: 0.218928 Best loss: 0.076569 Accuracy: 96.21% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=30, batch_size=100, learning_rate=0.02, total= 12.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1 0 Validation loss: 1.853634 Best loss: 1.853634 Accuracy: 18.73% 1 Validation loss: 1.868122 Best loss: 1.853634 Accuracy: 22.01% 2 Validation loss: 2.008039 Best loss: 1.853634 Accuracy: 19.27% 3 Validation loss: 2.204484 Best loss: 1.853634 Accuracy: 18.73% 4 Validation loss: 2.446456 Best loss: 1.853634 Accuracy: 18.73% 5 Validation loss: 2.247556 Best loss: 1.853634 Accuracy: 20.91% 6 Validation loss: 1.843468 Best loss: 1.843468 Accuracy: 20.91% 7 Validation loss: 2.451968 Best loss: 1.843468 Accuracy: 20.91% 8 Validation loss: 1.802150 Best loss: 1.802150 Accuracy: 19.08% 9 Validation loss: 3.539359 Best loss: 1.802150 Accuracy: 19.08% 10 Validation loss: 2.713006 Best loss: 1.802150 Accuracy: 19.27% 11 Validation loss: 2.944484 Best loss: 1.802150 Accuracy: 19.08% 12 Validation loss: 2.694592 Best loss: 1.802150 Accuracy: 22.01% 13 Validation loss: 2.691655 Best loss: 1.802150 Accuracy: 22.01% 14 Validation loss: 3.149468 Best loss: 1.802150 Accuracy: 18.73% 15 Validation loss: 3.263842 Best loss: 1.802150 Accuracy: 22.01% 16 Validation loss: 3.944899 Best loss: 1.802150 Accuracy: 18.73% 17 Validation loss: 3.008318 Best loss: 1.802150 Accuracy: 19.08% 18 Validation loss: 3.772084 Best loss: 1.802150 Accuracy: 20.91% 19 Validation loss: 4.505542 Best loss: 1.802150 Accuracy: 22.01% 20 Validation loss: 2.278224 Best loss: 1.802150 Accuracy: 20.91% 21 Validation loss: 2.641157 Best loss: 1.802150 Accuracy: 20.91% 22 Validation loss: 2.141643 Best loss: 1.802150 Accuracy: 22.01% 23 Validation loss: 3.155476 Best loss: 1.802150 Accuracy: 20.91% 24 Validation loss: 3.354372 Best loss: 1.802150 Accuracy: 20.91% 25 Validation loss: 3.570458 Best loss: 1.802150 Accuracy: 19.27% 26 Validation loss: 2.823988 Best loss: 1.802150 Accuracy: 20.91% 27 Validation loss: 1.909329 Best loss: 1.802150 Accuracy: 19.08% 28 Validation loss: 3.888405 Best loss: 1.802150 Accuracy: 19.08% 29 Validation loss: 1.805158 Best loss: 1.802150 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1, total= 1.8min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1 0 Validation loss: 2.229623 Best loss: 2.229623 Accuracy: 19.27% 1 Validation loss: 2.097598 Best loss: 2.097598 Accuracy: 19.27% 2 Validation loss: 1.913379 Best loss: 1.913379 Accuracy: 22.01% 3 Validation loss: 3.233124 Best loss: 1.913379 Accuracy: 22.01% 4 Validation loss: 2.211887 Best loss: 1.913379 Accuracy: 22.01% 5 Validation loss: 2.346187 Best loss: 1.913379 Accuracy: 20.91% 6 Validation loss: 3.233730 Best loss: 1.913379 Accuracy: 19.27% 7 Validation loss: 2.571963 Best loss: 1.913379 Accuracy: 20.91% 8 Validation loss: 1.956990 Best loss: 1.913379 Accuracy: 20.91% 9 Validation loss: 2.608446 Best loss: 1.913379 Accuracy: 22.01% 10 Validation loss: 3.699446 Best loss: 1.913379 Accuracy: 20.91% 11 Validation loss: 3.938284 Best loss: 1.913379 Accuracy: 19.08% 12 Validation loss: 1.985877 Best loss: 1.913379 Accuracy: 22.01% 13 Validation loss: 3.206213 Best loss: 1.913379 Accuracy: 20.91% 14 Validation loss: 2.733637 Best loss: 1.913379 Accuracy: 19.27% 15 Validation loss: 3.275279 Best loss: 1.913379 Accuracy: 20.91% 16 Validation loss: 2.141147 Best loss: 1.913379 Accuracy: 20.91% 17 Validation loss: 2.648915 Best loss: 1.913379 Accuracy: 20.91% 18 Validation loss: 2.909515 Best loss: 1.913379 Accuracy: 19.08% 19 Validation loss: 5.017917 Best loss: 1.913379 Accuracy: 18.73% 20 Validation loss: 2.628436 Best loss: 1.913379 Accuracy: 18.73% 21 Validation loss: 2.133545 Best loss: 1.913379 Accuracy: 19.27% 22 Validation loss: 3.064233 Best loss: 1.913379 Accuracy: 19.27% 23 Validation loss: 2.511384 Best loss: 1.913379 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1, total= 1.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1 0 Validation loss: 1.798246 Best loss: 1.798246 Accuracy: 20.91% 1 Validation loss: 2.379262 Best loss: 1.798246 Accuracy: 22.01% 2 Validation loss: 2.511294 Best loss: 1.798246 Accuracy: 19.08% 3 Validation loss: 3.318276 Best loss: 1.798246 Accuracy: 22.01% 4 Validation loss: 1.967625 Best loss: 1.798246 Accuracy: 19.27% 5 Validation loss: 2.513290 Best loss: 1.798246 Accuracy: 20.91% 6 Validation loss: 2.834153 Best loss: 1.798246 Accuracy: 19.27% 7 Validation loss: 3.146398 Best loss: 1.798246 Accuracy: 22.01% 8 Validation loss: 1.628950 Best loss: 1.628950 Accuracy: 18.73% 9 Validation loss: 2.255082 Best loss: 1.628950 Accuracy: 22.01% 10 Validation loss: 3.483264 Best loss: 1.628950 Accuracy: 19.27% 11 Validation loss: 2.788608 Best loss: 1.628950 Accuracy: 18.73% 12 Validation loss: 3.997707 Best loss: 1.628950 Accuracy: 18.73% 13 Validation loss: 2.768285 Best loss: 1.628950 Accuracy: 22.01% 14 Validation loss: 2.797261 Best loss: 1.628950 Accuracy: 20.91% 15 Validation loss: 1.709994 Best loss: 1.628950 Accuracy: 22.01% 16 Validation loss: 3.146127 Best loss: 1.628950 Accuracy: 20.91% 17 Validation loss: 2.609735 Best loss: 1.628950 Accuracy: 18.73% 18 Validation loss: 2.135784 Best loss: 1.628950 Accuracy: 18.73% 19 Validation loss: 3.607477 Best loss: 1.628950 Accuracy: 22.01% 20 Validation loss: 2.608031 Best loss: 1.628950 Accuracy: 19.27% 21 Validation loss: 1.980636 Best loss: 1.628950 Accuracy: 22.01% 22 Validation loss: 2.960027 Best loss: 1.628950 Accuracy: 22.01% 23 Validation loss: 2.321966 Best loss: 1.628950 Accuracy: 19.27% 24 Validation loss: 2.480272 Best loss: 1.628950 Accuracy: 22.01% 25 Validation loss: 1.935139 Best loss: 1.628950 Accuracy: 22.01% 26 Validation loss: 3.114526 Best loss: 1.628950 Accuracy: 18.73% 27 Validation loss: 3.026548 Best loss: 1.628950 Accuracy: 20.91% 28 Validation loss: 3.077613 Best loss: 1.628950 Accuracy: 19.27% 29 Validation loss: 5.017604 Best loss: 1.628950 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=10, learning_rate=0.1, total= 1.8min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1 0 Validation loss: 0.322626 Best loss: 0.322626 Accuracy: 90.23% 1 Validation loss: 0.198175 Best loss: 0.198175 Accuracy: 94.29% 2 Validation loss: 0.156252 Best loss: 0.156252 Accuracy: 95.54% 3 Validation loss: 0.156130 Best loss: 0.156130 Accuracy: 95.74% 4 Validation loss: 0.143108 Best loss: 0.143108 Accuracy: 95.97% 5 Validation loss: 0.167598 Best loss: 0.143108 Accuracy: 95.54% 6 Validation loss: 0.203577 Best loss: 0.143108 Accuracy: 94.14% 7 Validation loss: 0.116675 Best loss: 0.116675 Accuracy: 97.15% 8 Validation loss: 0.159621 Best loss: 0.116675 Accuracy: 95.15% 9 Validation loss: 0.180793 Best loss: 0.116675 Accuracy: 95.23% 10 Validation loss: 0.113504 Best loss: 0.113504 Accuracy: 96.91% 11 Validation loss: 1.620127 Best loss: 0.113504 Accuracy: 19.08% 12 Validation loss: 1.632742 Best loss: 0.113504 Accuracy: 19.27% 13 Validation loss: 1.611498 Best loss: 0.113504 Accuracy: 20.91% 14 Validation loss: 1.622743 Best loss: 0.113504 Accuracy: 19.08% 15 Validation loss: 1.640201 Best loss: 0.113504 Accuracy: 22.01% 16 Validation loss: 1.644194 Best loss: 0.113504 Accuracy: 20.91% 17 Validation loss: 1.653985 Best loss: 0.113504 Accuracy: 19.27% 18 Validation loss: 1.663153 Best loss: 0.113504 Accuracy: 18.73% 19 Validation loss: 1.667054 Best loss: 0.113504 Accuracy: 20.91% 20 Validation loss: 1.730998 Best loss: 0.113504 Accuracy: 22.01% 21 Validation loss: 1.625564 Best loss: 0.113504 Accuracy: 22.01% 22 Validation loss: 1.678629 Best loss: 0.113504 Accuracy: 22.01% 23 Validation loss: 1.630871 Best loss: 0.113504 Accuracy: 22.01% 24 Validation loss: 1.721025 Best loss: 0.113504 Accuracy: 19.27% 25 Validation loss: 1.709783 Best loss: 0.113504 Accuracy: 22.01% 26 Validation loss: 1.728928 Best loss: 0.113504 Accuracy: 19.08% 27 Validation loss: 1.658524 Best loss: 0.113504 Accuracy: 18.73% 28 Validation loss: 1.837201 Best loss: 0.113504 Accuracy: 22.01% 29 Validation loss: 1.627250 Best loss: 0.113504 Accuracy: 18.73% 30 Validation loss: 1.891822 Best loss: 0.113504 Accuracy: 19.27% 31 Validation loss: 1.768033 Best loss: 0.113504 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, total= 12.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1 0 Validation loss: 0.960785 Best loss: 0.960785 Accuracy: 61.85% 1 Validation loss: 0.369058 Best loss: 0.369058 Accuracy: 88.74% 2 Validation loss: 0.283855 Best loss: 0.283855 Accuracy: 91.75% 3 Validation loss: 0.217153 Best loss: 0.217153 Accuracy: 93.98% 4 Validation loss: 0.254724 Best loss: 0.217153 Accuracy: 93.12% 5 Validation loss: 0.264694 Best loss: 0.217153 Accuracy: 93.47% 6 Validation loss: 0.146291 Best loss: 0.146291 Accuracy: 96.40% 7 Validation loss: 0.169705 Best loss: 0.146291 Accuracy: 96.05% 8 Validation loss: 0.149521 Best loss: 0.146291 Accuracy: 96.25% 9 Validation loss: 1.675218 Best loss: 0.146291 Accuracy: 22.01% 10 Validation loss: 1.693192 Best loss: 0.146291 Accuracy: 20.91% 11 Validation loss: 1.618160 Best loss: 0.146291 Accuracy: 22.01% 12 Validation loss: 1.684494 Best loss: 0.146291 Accuracy: 19.27% 13 Validation loss: 1.662298 Best loss: 0.146291 Accuracy: 20.91% 14 Validation loss: 1.772415 Best loss: 0.146291 Accuracy: 22.01% 15 Validation loss: 1.716065 Best loss: 0.146291 Accuracy: 18.73% 16 Validation loss: 1.643608 Best loss: 0.146291 Accuracy: 22.01% 17 Validation loss: 1.707370 Best loss: 0.146291 Accuracy: 20.91% 18 Validation loss: 1.654873 Best loss: 0.146291 Accuracy: 22.01% 19 Validation loss: 1.645471 Best loss: 0.146291 Accuracy: 22.01% 20 Validation loss: 1.721684 Best loss: 0.146291 Accuracy: 20.91% 21 Validation loss: 1.771736 Best loss: 0.146291 Accuracy: 22.01% 22 Validation loss: 1.764241 Best loss: 0.146291 Accuracy: 18.73% 23 Validation loss: 1.728042 Best loss: 0.146291 Accuracy: 22.01% 24 Validation loss: 1.807453 Best loss: 0.146291 Accuracy: 18.73% 25 Validation loss: 1.769534 Best loss: 0.146291 Accuracy: 22.01% 26 Validation loss: 1.686877 Best loss: 0.146291 Accuracy: 20.91% 27 Validation loss: 1.970301 Best loss: 0.146291 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, total= 11.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1 0 Validation loss: 1.135957 Best loss: 1.135957 Accuracy: 68.65% 1 Validation loss: 0.288814 Best loss: 0.288814 Accuracy: 93.12% 2 Validation loss: 0.163132 Best loss: 0.163132 Accuracy: 96.36% 3 Validation loss: 0.224376 Best loss: 0.163132 Accuracy: 95.70% 4 Validation loss: 0.223569 Best loss: 0.163132 Accuracy: 95.86% 5 Validation loss: 0.188699 Best loss: 0.163132 Accuracy: 95.74% 6 Validation loss: 0.203973 Best loss: 0.163132 Accuracy: 96.17% 7 Validation loss: 0.171648 Best loss: 0.163132 Accuracy: 96.56% 8 Validation loss: 0.212064 Best loss: 0.163132 Accuracy: 95.82% 9 Validation loss: 0.198122 Best loss: 0.163132 Accuracy: 95.78% 10 Validation loss: 0.143920 Best loss: 0.143920 Accuracy: 96.91% 11 Validation loss: 0.135878 Best loss: 0.135878 Accuracy: 97.38% 12 Validation loss: 1.683018 Best loss: 0.135878 Accuracy: 20.91% 13 Validation loss: 1.643626 Best loss: 0.135878 Accuracy: 19.08% 14 Validation loss: 1.641420 Best loss: 0.135878 Accuracy: 19.08% 15 Validation loss: 1.702199 Best loss: 0.135878 Accuracy: 20.91% 16 Validation loss: 1.663595 Best loss: 0.135878 Accuracy: 18.73% 17 Validation loss: 1.645927 Best loss: 0.135878 Accuracy: 18.73% 18 Validation loss: 1.697956 Best loss: 0.135878 Accuracy: 18.73% 19 Validation loss: 1.650169 Best loss: 0.135878 Accuracy: 18.73% 20 Validation loss: 1.628427 Best loss: 0.135878 Accuracy: 20.91% 21 Validation loss: 1.732560 Best loss: 0.135878 Accuracy: 18.73% 22 Validation loss: 1.650133 Best loss: 0.135878 Accuracy: 19.08% 23 Validation loss: 1.726199 Best loss: 0.135878 Accuracy: 22.01% 24 Validation loss: 1.676762 Best loss: 0.135878 Accuracy: 20.91% 25 Validation loss: 1.698555 Best loss: 0.135878 Accuracy: 18.73% 26 Validation loss: 1.641431 Best loss: 0.135878 Accuracy: 22.01% 27 Validation loss: 1.706518 Best loss: 0.135878 Accuracy: 20.91% 28 Validation loss: 1.732833 Best loss: 0.135878 Accuracy: 20.91% 29 Validation loss: 1.656679 Best loss: 0.135878 Accuracy: 22.01% 30 Validation loss: 1.722130 Best loss: 0.135878 Accuracy: 20.91% 31 Validation loss: 1.824034 Best loss: 0.135878 Accuracy: 18.73% 32 Validation loss: 1.620046 Best loss: 0.135878 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, total= 13.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 1.917454 Best loss: 1.917454 Accuracy: 24.59% 1 Validation loss: 1.432599 Best loss: 1.432599 Accuracy: 43.71% 2 Validation loss: 1.012935 Best loss: 1.012935 Accuracy: 60.05% 3 Validation loss: 0.660141 Best loss: 0.660141 Accuracy: 74.35% 4 Validation loss: 0.503325 Best loss: 0.503325 Accuracy: 86.86% 5 Validation loss: 0.543369 Best loss: 0.503325 Accuracy: 85.38% 6 Validation loss: 0.315726 Best loss: 0.315726 Accuracy: 90.97% 7 Validation loss: 0.312311 Best loss: 0.312311 Accuracy: 90.46% 8 Validation loss: 0.261455 Best loss: 0.261455 Accuracy: 92.57% 9 Validation loss: 0.292165 Best loss: 0.261455 Accuracy: 91.24% 10 Validation loss: 0.208687 Best loss: 0.208687 Accuracy: 94.14% 11 Validation loss: 0.206743 Best loss: 0.206743 Accuracy: 94.25% 12 Validation loss: 0.160423 Best loss: 0.160423 Accuracy: 95.35% 13 Validation loss: 2.946149 Best loss: 0.160423 Accuracy: 41.28% 14 Validation loss: 3084.905762 Best loss: 0.160423 Accuracy: 19.08% 15 Validation loss: 9248.495117 Best loss: 0.160423 Accuracy: 20.91% 16 Validation loss: 26261.367188 Best loss: 0.160423 Accuracy: 19.08% 17 Validation loss: 6138.978516 Best loss: 0.160423 Accuracy: 20.91% 18 Validation loss: 3672.275879 Best loss: 0.160423 Accuracy: 19.08% 19 Validation loss: 4021.585693 Best loss: 0.160423 Accuracy: 30.57% 20 Validation loss: 612.339661 Best loss: 0.160423 Accuracy: 20.48% 21 Validation loss: 872.898926 Best loss: 0.160423 Accuracy: 21.15% 22 Validation loss: 225.453629 Best loss: 0.160423 Accuracy: 48.91% 23 Validation loss: 366.915344 Best loss: 0.160423 Accuracy: 35.30% 24 Validation loss: 301.095917 Best loss: 0.160423 Accuracy: 45.11% 25 Validation loss: 270.264374 Best loss: 0.160423 Accuracy: 38.35% 26 Validation loss: 147.714493 Best loss: 0.160423 Accuracy: 46.01% 27 Validation loss: 206.943680 Best loss: 0.160423 Accuracy: 44.92% 28 Validation loss: 208.675888 Best loss: 0.160423 Accuracy: 41.40% 29 Validation loss: 134.975739 Best loss: 0.160423 Accuracy: 48.79% 30 Validation loss: 176.032074 Best loss: 0.160423 Accuracy: 50.74% 31 Validation loss: 136.624344 Best loss: 0.160423 Accuracy: 53.91% 32 Validation loss: 104.821846 Best loss: 0.160423 Accuracy: 47.30% 33 Validation loss: 157.418839 Best loss: 0.160423 Accuracy: 51.99% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 5.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 1.254347 Best loss: 1.254347 Accuracy: 44.76% 1 Validation loss: 0.496946 Best loss: 0.496946 Accuracy: 85.18% 2 Validation loss: 0.539125 Best loss: 0.496946 Accuracy: 81.35% 3 Validation loss: 121.926331 Best loss: 0.496946 Accuracy: 22.01% 4 Validation loss: 173.366699 Best loss: 0.496946 Accuracy: 15.25% 5 Validation loss: 1107.477661 Best loss: 0.496946 Accuracy: 18.73% 6 Validation loss: 111.301514 Best loss: 0.496946 Accuracy: 19.98% 7 Validation loss: 127.757462 Best loss: 0.496946 Accuracy: 26.62% 8 Validation loss: 19.767872 Best loss: 0.496946 Accuracy: 47.22% 9 Validation loss: 3.386102 Best loss: 0.496946 Accuracy: 63.45% 10 Validation loss: 1.791267 Best loss: 0.496946 Accuracy: 62.94% 11 Validation loss: 1.250653 Best loss: 0.496946 Accuracy: 66.89% 12 Validation loss: 1.427207 Best loss: 0.496946 Accuracy: 60.75% 13 Validation loss: 2.899854 Best loss: 0.496946 Accuracy: 54.57% 14 Validation loss: 0.966314 Best loss: 0.496946 Accuracy: 72.63% 15 Validation loss: 0.847541 Best loss: 0.496946 Accuracy: 73.46% 16 Validation loss: 0.734963 Best loss: 0.496946 Accuracy: 76.54% 17 Validation loss: 0.810539 Best loss: 0.496946 Accuracy: 74.08% 18 Validation loss: 0.709540 Best loss: 0.496946 Accuracy: 76.31% 19 Validation loss: 0.689016 Best loss: 0.496946 Accuracy: 77.64% 20 Validation loss: 0.747442 Best loss: 0.496946 Accuracy: 74.32% 21 Validation loss: 0.671759 Best loss: 0.496946 Accuracy: 77.95% 22 Validation loss: 0.604438 Best loss: 0.496946 Accuracy: 79.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 3.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1 0 Validation loss: 1.194224 Best loss: 1.194224 Accuracy: 52.35% 1 Validation loss: 0.805726 Best loss: 0.805726 Accuracy: 66.58% 2 Validation loss: 0.538406 Best loss: 0.538406 Accuracy: 80.14% 3 Validation loss: 0.468543 Best loss: 0.468543 Accuracy: 84.60% 4 Validation loss: 0.309620 Best loss: 0.309620 Accuracy: 90.93% 5 Validation loss: 0.303060 Best loss: 0.303060 Accuracy: 91.28% 6 Validation loss: 0.373308 Best loss: 0.303060 Accuracy: 91.91% 7 Validation loss: 0.227810 Best loss: 0.227810 Accuracy: 93.51% 8 Validation loss: 0.273122 Best loss: 0.227810 Accuracy: 92.53% 9 Validation loss: 0.194995 Best loss: 0.194995 Accuracy: 94.37% 10 Validation loss: 9.281303 Best loss: 0.194995 Accuracy: 23.77% 11 Validation loss: 3721.935791 Best loss: 0.194995 Accuracy: 2.93% 12 Validation loss: 23966.224609 Best loss: 0.194995 Accuracy: 18.80% 13 Validation loss: 20904.236328 Best loss: 0.194995 Accuracy: 20.91% 14 Validation loss: 1414.141479 Best loss: 0.194995 Accuracy: 40.15% 15 Validation loss: 758.466675 Best loss: 0.194995 Accuracy: 36.36% 16 Validation loss: 176.688385 Best loss: 0.194995 Accuracy: 48.01% 17 Validation loss: 93.105270 Best loss: 0.194995 Accuracy: 51.99% 18 Validation loss: 179.335922 Best loss: 0.194995 Accuracy: 42.30% 19 Validation loss: 98.226852 Best loss: 0.194995 Accuracy: 40.15% 20 Validation loss: 103.828751 Best loss: 0.194995 Accuracy: 53.21% 21 Validation loss: 141.663940 Best loss: 0.194995 Accuracy: 46.17% 22 Validation loss: 54.845154 Best loss: 0.194995 Accuracy: 53.71% 23 Validation loss: 53.296852 Best loss: 0.194995 Accuracy: 61.34% 24 Validation loss: 34.111702 Best loss: 0.194995 Accuracy: 52.81% 25 Validation loss: 205.660156 Best loss: 0.194995 Accuracy: 19.04% 26 Validation loss: 34.377480 Best loss: 0.194995 Accuracy: 51.60% 27 Validation loss: 38.031021 Best loss: 0.194995 Accuracy: 50.78% 28 Validation loss: 34.402336 Best loss: 0.194995 Accuracy: 52.58% 29 Validation loss: 19.557745 Best loss: 0.194995 Accuracy: 50.55% 30 Validation loss: 29.456253 Best loss: 0.194995 Accuracy: 54.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=50, batch_size=500, learning_rate=0.1, total= 4.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02 0 Validation loss: 0.229409 Best loss: 0.229409 Accuracy: 93.28% 1 Validation loss: 0.129947 Best loss: 0.129947 Accuracy: 96.05% 2 Validation loss: 0.100922 Best loss: 0.100922 Accuracy: 97.15% 3 Validation loss: 0.085018 Best loss: 0.085018 Accuracy: 97.15% 4 Validation loss: 0.092292 Best loss: 0.085018 Accuracy: 97.22% 5 Validation loss: 0.080592 Best loss: 0.080592 Accuracy: 97.77% 6 Validation loss: 0.063582 Best loss: 0.063582 Accuracy: 98.08% 7 Validation loss: 0.074141 Best loss: 0.063582 Accuracy: 97.73% 8 Validation loss: 0.069194 Best loss: 0.063582 Accuracy: 97.85% 9 Validation loss: 0.067778 Best loss: 0.063582 Accuracy: 98.28% 10 Validation loss: 0.071136 Best loss: 0.063582 Accuracy: 98.20% 11 Validation loss: 0.086366 Best loss: 0.063582 Accuracy: 97.89% 12 Validation loss: 0.069214 Best loss: 0.063582 Accuracy: 97.93% 13 Validation loss: 0.059420 Best loss: 0.059420 Accuracy: 98.48% 14 Validation loss: 0.082708 Best loss: 0.059420 Accuracy: 97.93% 15 Validation loss: 0.054529 Best loss: 0.054529 Accuracy: 98.71% 16 Validation loss: 0.067361 Best loss: 0.054529 Accuracy: 98.63% 17 Validation loss: 0.054728 Best loss: 0.054529 Accuracy: 98.75% 18 Validation loss: 0.055371 Best loss: 0.054529 Accuracy: 98.79% 19 Validation loss: 0.058372 Best loss: 0.054529 Accuracy: 98.87% 20 Validation loss: 0.073798 Best loss: 0.054529 Accuracy: 98.79% 21 Validation loss: 0.090309 Best loss: 0.054529 Accuracy: 98.44% 22 Validation loss: 0.090133 Best loss: 0.054529 Accuracy: 98.67% 23 Validation loss: 0.079768 Best loss: 0.054529 Accuracy: 98.20% 24 Validation loss: 0.054613 Best loss: 0.054529 Accuracy: 98.67% 25 Validation loss: 0.075571 Best loss: 0.054529 Accuracy: 98.79% 26 Validation loss: 0.094726 Best loss: 0.054529 Accuracy: 98.32% 27 Validation loss: 0.075905 Best loss: 0.054529 Accuracy: 98.71% 28 Validation loss: 0.071283 Best loss: 0.054529 Accuracy: 98.48% 29 Validation loss: 0.083590 Best loss: 0.054529 Accuracy: 98.87% 30 Validation loss: 0.081415 Best loss: 0.054529 Accuracy: 98.55% 31 Validation loss: 0.083469 Best loss: 0.054529 Accuracy: 98.67% 32 Validation loss: 0.097219 Best loss: 0.054529 Accuracy: 98.48% 33 Validation loss: 0.097422 Best loss: 0.054529 Accuracy: 98.63% 34 Validation loss: 0.102108 Best loss: 0.054529 Accuracy: 98.48% 35 Validation loss: 0.098452 Best loss: 0.054529 Accuracy: 97.69% 36 Validation loss: 0.092863 Best loss: 0.054529 Accuracy: 98.32% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02, total= 4.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02 0 Validation loss: 0.174298 Best loss: 0.174298 Accuracy: 95.07% 1 Validation loss: 0.090371 Best loss: 0.090371 Accuracy: 97.30% 2 Validation loss: 0.079053 Best loss: 0.079053 Accuracy: 97.34% 3 Validation loss: 0.074898 Best loss: 0.074898 Accuracy: 97.69% 4 Validation loss: 0.086567 Best loss: 0.074898 Accuracy: 97.73% 5 Validation loss: 0.063863 Best loss: 0.063863 Accuracy: 97.73% 6 Validation loss: 0.058720 Best loss: 0.058720 Accuracy: 98.20% 7 Validation loss: 0.049724 Best loss: 0.049724 Accuracy: 98.36% 8 Validation loss: 0.058220 Best loss: 0.049724 Accuracy: 98.36% 9 Validation loss: 0.046974 Best loss: 0.046974 Accuracy: 98.40% 10 Validation loss: 0.046539 Best loss: 0.046539 Accuracy: 98.44% 11 Validation loss: 0.063236 Best loss: 0.046539 Accuracy: 98.44% 12 Validation loss: 0.050568 Best loss: 0.046539 Accuracy: 98.67% 13 Validation loss: 0.054750 Best loss: 0.046539 Accuracy: 98.71% 14 Validation loss: 0.079764 Best loss: 0.046539 Accuracy: 98.55% 15 Validation loss: 0.054522 Best loss: 0.046539 Accuracy: 99.02% 16 Validation loss: 0.088934 Best loss: 0.046539 Accuracy: 98.55% 17 Validation loss: 0.061068 Best loss: 0.046539 Accuracy: 98.59% 18 Validation loss: 0.066996 Best loss: 0.046539 Accuracy: 98.44% 19 Validation loss: 0.056973 Best loss: 0.046539 Accuracy: 98.32% 20 Validation loss: 0.047212 Best loss: 0.046539 Accuracy: 98.98% 21 Validation loss: 0.045249 Best loss: 0.045249 Accuracy: 98.98% 22 Validation loss: 0.058381 Best loss: 0.045249 Accuracy: 98.91% 23 Validation loss: 0.072985 Best loss: 0.045249 Accuracy: 98.79% 24 Validation loss: 0.091483 Best loss: 0.045249 Accuracy: 98.40% 25 Validation loss: 0.069902 Best loss: 0.045249 Accuracy: 98.83% 26 Validation loss: 0.052421 Best loss: 0.045249 Accuracy: 99.02% 27 Validation loss: 0.078245 Best loss: 0.045249 Accuracy: 98.55% 28 Validation loss: 0.067997 Best loss: 0.045249 Accuracy: 98.55% 29 Validation loss: 0.061625 Best loss: 0.045249 Accuracy: 98.91% 30 Validation loss: 0.082193 Best loss: 0.045249 Accuracy: 98.36% 31 Validation loss: 0.064793 Best loss: 0.045249 Accuracy: 99.06% 32 Validation loss: 0.066582 Best loss: 0.045249 Accuracy: 98.71% 33 Validation loss: 0.061991 Best loss: 0.045249 Accuracy: 98.98% 34 Validation loss: 0.052146 Best loss: 0.045249 Accuracy: 98.94% 35 Validation loss: 0.077598 Best loss: 0.045249 Accuracy: 98.83% 36 Validation loss: 0.082148 Best loss: 0.045249 Accuracy: 98.75% 37 Validation loss: 0.053539 Best loss: 0.045249 Accuracy: 98.79% 38 Validation loss: 0.060247 Best loss: 0.045249 Accuracy: 98.79% 39 Validation loss: 0.076902 Best loss: 0.045249 Accuracy: 98.63% 40 Validation loss: 680.095947 Best loss: 0.045249 Accuracy: 20.91% 41 Validation loss: 1.920724 Best loss: 0.045249 Accuracy: 19.08% 42 Validation loss: 1.613273 Best loss: 0.045249 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02, total= 5.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02 0 Validation loss: 0.179913 Best loss: 0.179913 Accuracy: 93.82% 1 Validation loss: 0.086437 Best loss: 0.086437 Accuracy: 97.26% 2 Validation loss: 0.072516 Best loss: 0.072516 Accuracy: 97.81% 3 Validation loss: 0.077930 Best loss: 0.072516 Accuracy: 97.93% 4 Validation loss: 0.057668 Best loss: 0.057668 Accuracy: 98.16% 5 Validation loss: 0.061281 Best loss: 0.057668 Accuracy: 98.44% 6 Validation loss: 0.078786 Best loss: 0.057668 Accuracy: 97.89% 7 Validation loss: 0.067796 Best loss: 0.057668 Accuracy: 98.24% 8 Validation loss: 0.057618 Best loss: 0.057618 Accuracy: 98.55% 9 Validation loss: 0.053576 Best loss: 0.053576 Accuracy: 98.63% 10 Validation loss: 0.088743 Best loss: 0.053576 Accuracy: 97.93% 11 Validation loss: 0.120267 Best loss: 0.053576 Accuracy: 98.67% 12 Validation loss: 0.088222 Best loss: 0.053576 Accuracy: 98.32% 13 Validation loss: 0.065651 Best loss: 0.053576 Accuracy: 98.55% 14 Validation loss: 0.068698 Best loss: 0.053576 Accuracy: 98.48% 15 Validation loss: 0.076104 Best loss: 0.053576 Accuracy: 98.48% 16 Validation loss: 0.078266 Best loss: 0.053576 Accuracy: 98.63% 17 Validation loss: 0.055749 Best loss: 0.053576 Accuracy: 98.94% 18 Validation loss: 0.059635 Best loss: 0.053576 Accuracy: 98.87% 19 Validation loss: 0.065766 Best loss: 0.053576 Accuracy: 98.98% 20 Validation loss: 0.077436 Best loss: 0.053576 Accuracy: 98.94% 21 Validation loss: 0.065620 Best loss: 0.053576 Accuracy: 98.67% 22 Validation loss: 0.054288 Best loss: 0.053576 Accuracy: 98.91% 23 Validation loss: 0.062137 Best loss: 0.053576 Accuracy: 99.06% 24 Validation loss: 0.060430 Best loss: 0.053576 Accuracy: 99.02% 25 Validation loss: 0.079974 Best loss: 0.053576 Accuracy: 98.87% 26 Validation loss: 0.089659 Best loss: 0.053576 Accuracy: 98.67% 27 Validation loss: 0.077419 Best loss: 0.053576 Accuracy: 98.75% 28 Validation loss: 0.074856 Best loss: 0.053576 Accuracy: 98.75% 29 Validation loss: 0.083730 Best loss: 0.053576 Accuracy: 98.48% 30 Validation loss: 0.077704 Best loss: 0.053576 Accuracy: 98.51% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.02, total= 3.8s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01 0 Validation loss: 0.089220 Best loss: 0.089220 Accuracy: 97.50% 1 Validation loss: 0.077197 Best loss: 0.077197 Accuracy: 97.81% 2 Validation loss: 0.099691 Best loss: 0.077197 Accuracy: 97.30% 3 Validation loss: 0.050653 Best loss: 0.050653 Accuracy: 98.44% 4 Validation loss: 0.070010 Best loss: 0.050653 Accuracy: 98.01% 5 Validation loss: 0.090803 Best loss: 0.050653 Accuracy: 97.62% 6 Validation loss: 0.072265 Best loss: 0.050653 Accuracy: 97.89% 7 Validation loss: 0.129768 Best loss: 0.050653 Accuracy: 98.01% 8 Validation loss: 0.062603 Best loss: 0.050653 Accuracy: 98.24% 9 Validation loss: 0.089818 Best loss: 0.050653 Accuracy: 98.32% 10 Validation loss: 0.059601 Best loss: 0.050653 Accuracy: 98.51% 11 Validation loss: 0.060324 Best loss: 0.050653 Accuracy: 98.55% 12 Validation loss: 0.058772 Best loss: 0.050653 Accuracy: 98.48% 13 Validation loss: 0.066730 Best loss: 0.050653 Accuracy: 98.48% 14 Validation loss: 0.065795 Best loss: 0.050653 Accuracy: 98.55% 15 Validation loss: 0.069129 Best loss: 0.050653 Accuracy: 98.44% 16 Validation loss: 0.058333 Best loss: 0.050653 Accuracy: 98.67% 17 Validation loss: 0.064848 Best loss: 0.050653 Accuracy: 98.63% 18 Validation loss: 0.057693 Best loss: 0.050653 Accuracy: 98.63% 19 Validation loss: 0.083892 Best loss: 0.050653 Accuracy: 97.81% 20 Validation loss: 0.087547 Best loss: 0.050653 Accuracy: 98.40% 21 Validation loss: 0.062632 Best loss: 0.050653 Accuracy: 98.75% 22 Validation loss: 0.066252 Best loss: 0.050653 Accuracy: 98.75% 23 Validation loss: 0.076768 Best loss: 0.050653 Accuracy: 98.71% 24 Validation loss: 0.052061 Best loss: 0.050653 Accuracy: 98.91% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01, total= 10.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01 0 Validation loss: 0.107760 Best loss: 0.107760 Accuracy: 96.64% 1 Validation loss: 0.080314 Best loss: 0.080314 Accuracy: 97.69% 2 Validation loss: 0.069165 Best loss: 0.069165 Accuracy: 97.93% 3 Validation loss: 0.054196 Best loss: 0.054196 Accuracy: 98.40% 4 Validation loss: 0.069517 Best loss: 0.054196 Accuracy: 98.01% 5 Validation loss: 0.054957 Best loss: 0.054196 Accuracy: 98.44% 6 Validation loss: 0.076731 Best loss: 0.054196 Accuracy: 97.89% 7 Validation loss: 0.067566 Best loss: 0.054196 Accuracy: 98.20% 8 Validation loss: 0.064683 Best loss: 0.054196 Accuracy: 98.40% 9 Validation loss: 0.071233 Best loss: 0.054196 Accuracy: 98.24% 10 Validation loss: 0.066205 Best loss: 0.054196 Accuracy: 98.32% 11 Validation loss: 0.066348 Best loss: 0.054196 Accuracy: 98.48% 12 Validation loss: 0.098462 Best loss: 0.054196 Accuracy: 97.85% 13 Validation loss: 0.079011 Best loss: 0.054196 Accuracy: 98.32% 14 Validation loss: 0.062693 Best loss: 0.054196 Accuracy: 98.55% 15 Validation loss: 0.065690 Best loss: 0.054196 Accuracy: 98.55% 16 Validation loss: 0.068700 Best loss: 0.054196 Accuracy: 98.55% 17 Validation loss: 0.097693 Best loss: 0.054196 Accuracy: 97.93% 18 Validation loss: 0.107684 Best loss: 0.054196 Accuracy: 98.08% 19 Validation loss: 0.124029 Best loss: 0.054196 Accuracy: 97.93% 20 Validation loss: 0.096248 Best loss: 0.054196 Accuracy: 98.40% 21 Validation loss: 0.080599 Best loss: 0.054196 Accuracy: 98.51% 22 Validation loss: 0.085319 Best loss: 0.054196 Accuracy: 98.44% 23 Validation loss: 0.092082 Best loss: 0.054196 Accuracy: 98.63% 24 Validation loss: 0.092587 Best loss: 0.054196 Accuracy: 98.20% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01, total= 10.6s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01 0 Validation loss: 0.098324 Best loss: 0.098324 Accuracy: 96.99% 1 Validation loss: 0.092978 Best loss: 0.092978 Accuracy: 97.30% 2 Validation loss: 0.079353 Best loss: 0.079353 Accuracy: 97.93% 3 Validation loss: 0.077526 Best loss: 0.077526 Accuracy: 97.85% 4 Validation loss: 0.065607 Best loss: 0.065607 Accuracy: 98.48% 5 Validation loss: 0.059822 Best loss: 0.059822 Accuracy: 98.24% 6 Validation loss: 0.057551 Best loss: 0.057551 Accuracy: 98.48% 7 Validation loss: 0.058445 Best loss: 0.057551 Accuracy: 98.36% 8 Validation loss: 0.060660 Best loss: 0.057551 Accuracy: 98.71% 9 Validation loss: 0.092340 Best loss: 0.057551 Accuracy: 97.85% 10 Validation loss: 0.072435 Best loss: 0.057551 Accuracy: 98.40% 11 Validation loss: 0.071178 Best loss: 0.057551 Accuracy: 98.32% 12 Validation loss: 0.090600 Best loss: 0.057551 Accuracy: 98.05% 13 Validation loss: 0.072256 Best loss: 0.057551 Accuracy: 98.36% 14 Validation loss: 0.063353 Best loss: 0.057551 Accuracy: 98.71% 15 Validation loss: 0.063890 Best loss: 0.057551 Accuracy: 98.91% 16 Validation loss: 0.074101 Best loss: 0.057551 Accuracy: 98.71% 17 Validation loss: 0.058423 Best loss: 0.057551 Accuracy: 98.83% 18 Validation loss: 0.070752 Best loss: 0.057551 Accuracy: 98.91% 19 Validation loss: 0.069582 Best loss: 0.057551 Accuracy: 98.71% 20 Validation loss: 0.078232 Best loss: 0.057551 Accuracy: 98.40% 21 Validation loss: 0.112293 Best loss: 0.057551 Accuracy: 98.32% 22 Validation loss: 0.092560 Best loss: 0.057551 Accuracy: 98.36% 23 Validation loss: 0.098389 Best loss: 0.057551 Accuracy: 98.79% 24 Validation loss: 0.114414 Best loss: 0.057551 Accuracy: 98.28% 25 Validation loss: 0.092485 Best loss: 0.057551 Accuracy: 98.20% 26 Validation loss: 0.080822 Best loss: 0.057551 Accuracy: 98.87% 27 Validation loss: 0.093992 Best loss: 0.057551 Accuracy: 98.75% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=100, learning_rate=0.01, total= 11.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05 0 Validation loss: 0.407585 Best loss: 0.407585 Accuracy: 89.25% 1 Validation loss: 0.150171 Best loss: 0.150171 Accuracy: 95.66% 2 Validation loss: 0.117681 Best loss: 0.117681 Accuracy: 96.60% 3 Validation loss: 0.090731 Best loss: 0.090731 Accuracy: 97.65% 4 Validation loss: 0.083879 Best loss: 0.083879 Accuracy: 97.69% 5 Validation loss: 0.093132 Best loss: 0.083879 Accuracy: 97.50% 6 Validation loss: 0.097599 Best loss: 0.083879 Accuracy: 97.22% 7 Validation loss: 0.093268 Best loss: 0.083879 Accuracy: 97.34% 8 Validation loss: 0.085936 Best loss: 0.083879 Accuracy: 97.69% 9 Validation loss: 0.082373 Best loss: 0.082373 Accuracy: 98.05% 10 Validation loss: 0.089866 Best loss: 0.082373 Accuracy: 97.22% 11 Validation loss: 0.084928 Best loss: 0.082373 Accuracy: 97.62% 12 Validation loss: 0.084158 Best loss: 0.082373 Accuracy: 97.93% 13 Validation loss: 0.077944 Best loss: 0.077944 Accuracy: 97.89% 14 Validation loss: 0.071117 Best loss: 0.071117 Accuracy: 97.93% 15 Validation loss: 0.081678 Best loss: 0.071117 Accuracy: 98.01% 16 Validation loss: 0.090764 Best loss: 0.071117 Accuracy: 97.77% 17 Validation loss: 0.095537 Best loss: 0.071117 Accuracy: 97.62% 18 Validation loss: 0.095906 Best loss: 0.071117 Accuracy: 97.65% 19 Validation loss: 0.091024 Best loss: 0.071117 Accuracy: 98.08% 20 Validation loss: 0.127789 Best loss: 0.071117 Accuracy: 97.50% 21 Validation loss: 0.066878 Best loss: 0.066878 Accuracy: 98.08% 22 Validation loss: 0.077819 Best loss: 0.066878 Accuracy: 98.12% 23 Validation loss: 0.082964 Best loss: 0.066878 Accuracy: 98.16% 24 Validation loss: 0.135646 Best loss: 0.066878 Accuracy: 97.22% 25 Validation loss: 0.082845 Best loss: 0.066878 Accuracy: 97.62% 26 Validation loss: 0.072161 Best loss: 0.066878 Accuracy: 97.81% 27 Validation loss: 0.090639 Best loss: 0.066878 Accuracy: 98.08% 28 Validation loss: 0.079794 Best loss: 0.066878 Accuracy: 98.20% 29 Validation loss: 0.086298 Best loss: 0.066878 Accuracy: 98.36% 30 Validation loss: 0.094677 Best loss: 0.066878 Accuracy: 98.32% 31 Validation loss: 0.111512 Best loss: 0.066878 Accuracy: 97.73% 32 Validation loss: 0.078629 Best loss: 0.066878 Accuracy: 98.20% 33 Validation loss: 0.102015 Best loss: 0.066878 Accuracy: 98.36% 34 Validation loss: 0.133971 Best loss: 0.066878 Accuracy: 97.69% 35 Validation loss: 0.095556 Best loss: 0.066878 Accuracy: 98.24% 36 Validation loss: 0.086671 Best loss: 0.066878 Accuracy: 98.32% 37 Validation loss: 0.111889 Best loss: 0.066878 Accuracy: 97.73% 38 Validation loss: 0.077205 Best loss: 0.066878 Accuracy: 98.55% 39 Validation loss: 0.169527 Best loss: 0.066878 Accuracy: 98.44% 40 Validation loss: 0.102692 Best loss: 0.066878 Accuracy: 98.40% 41 Validation loss: 0.111560 Best loss: 0.066878 Accuracy: 98.40% 42 Validation loss: 0.110100 Best loss: 0.066878 Accuracy: 98.05% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05, total= 6.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05 0 Validation loss: 0.240096 Best loss: 0.240096 Accuracy: 93.75% 1 Validation loss: 0.114902 Best loss: 0.114902 Accuracy: 96.56% 2 Validation loss: 0.092681 Best loss: 0.092681 Accuracy: 97.46% 3 Validation loss: 0.088712 Best loss: 0.088712 Accuracy: 97.38% 4 Validation loss: 0.082413 Best loss: 0.082413 Accuracy: 97.65% 5 Validation loss: 0.094335 Best loss: 0.082413 Accuracy: 97.69% 6 Validation loss: 0.095968 Best loss: 0.082413 Accuracy: 97.85% 7 Validation loss: 0.070960 Best loss: 0.070960 Accuracy: 97.93% 8 Validation loss: 0.108825 Best loss: 0.070960 Accuracy: 97.07% 9 Validation loss: 0.072693 Best loss: 0.070960 Accuracy: 97.97% 10 Validation loss: 0.082912 Best loss: 0.070960 Accuracy: 97.73% 11 Validation loss: 0.070430 Best loss: 0.070430 Accuracy: 97.97% 12 Validation loss: 0.079514 Best loss: 0.070430 Accuracy: 98.12% 13 Validation loss: 0.088043 Best loss: 0.070430 Accuracy: 97.97% 14 Validation loss: 0.098412 Best loss: 0.070430 Accuracy: 98.20% 15 Validation loss: 0.083539 Best loss: 0.070430 Accuracy: 98.40% 16 Validation loss: 0.086173 Best loss: 0.070430 Accuracy: 97.77% 17 Validation loss: 0.082773 Best loss: 0.070430 Accuracy: 98.08% 18 Validation loss: 0.107361 Best loss: 0.070430 Accuracy: 97.93% 19 Validation loss: 0.100535 Best loss: 0.070430 Accuracy: 97.69% 20 Validation loss: 0.103552 Best loss: 0.070430 Accuracy: 97.62% 21 Validation loss: 0.094113 Best loss: 0.070430 Accuracy: 97.77% 22 Validation loss: 0.121308 Best loss: 0.070430 Accuracy: 97.26% 23 Validation loss: 0.095986 Best loss: 0.070430 Accuracy: 98.20% 24 Validation loss: 0.094551 Best loss: 0.070430 Accuracy: 97.93% 25 Validation loss: 0.094813 Best loss: 0.070430 Accuracy: 98.01% 26 Validation loss: 0.160824 Best loss: 0.070430 Accuracy: 97.97% 27 Validation loss: 102577.367188 Best loss: 0.070430 Accuracy: 19.27% 28 Validation loss: 2909.480957 Best loss: 0.070430 Accuracy: 51.72% 29 Validation loss: 171.516083 Best loss: 0.070430 Accuracy: 82.99% 30 Validation loss: 135.521774 Best loss: 0.070430 Accuracy: 82.60% 31 Validation loss: 88.530052 Best loss: 0.070430 Accuracy: 86.32% 32 Validation loss: 76.534668 Best loss: 0.070430 Accuracy: 86.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05, total= 4.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05 0 Validation loss: 0.506698 Best loss: 0.506698 Accuracy: 84.64% 1 Validation loss: 0.142504 Best loss: 0.142504 Accuracy: 96.09% 2 Validation loss: 0.140894 Best loss: 0.140894 Accuracy: 96.29% 3 Validation loss: 0.104456 Best loss: 0.104456 Accuracy: 97.19% 4 Validation loss: 0.099782 Best loss: 0.099782 Accuracy: 97.50% 5 Validation loss: 0.115304 Best loss: 0.099782 Accuracy: 97.26% 6 Validation loss: 0.106800 Best loss: 0.099782 Accuracy: 97.65% 7 Validation loss: 0.116525 Best loss: 0.099782 Accuracy: 97.26% 8 Validation loss: 0.109438 Best loss: 0.099782 Accuracy: 97.03% 9 Validation loss: 0.082539 Best loss: 0.082539 Accuracy: 97.58% 10 Validation loss: 0.086970 Best loss: 0.082539 Accuracy: 97.54% 11 Validation loss: 0.096180 Best loss: 0.082539 Accuracy: 97.34% 12 Validation loss: 0.086466 Best loss: 0.082539 Accuracy: 97.42% 13 Validation loss: 0.083519 Best loss: 0.082539 Accuracy: 97.81% 14 Validation loss: 0.103063 Best loss: 0.082539 Accuracy: 97.65% 15 Validation loss: 0.076208 Best loss: 0.076208 Accuracy: 97.97% 16 Validation loss: 0.086162 Best loss: 0.076208 Accuracy: 97.89% 17 Validation loss: 0.081447 Best loss: 0.076208 Accuracy: 97.97% 18 Validation loss: 0.055434 Best loss: 0.055434 Accuracy: 98.55% 19 Validation loss: 0.116258 Best loss: 0.055434 Accuracy: 97.07% 20 Validation loss: 0.085095 Best loss: 0.055434 Accuracy: 98.01% 21 Validation loss: 0.093456 Best loss: 0.055434 Accuracy: 97.85% 22 Validation loss: 0.091369 Best loss: 0.055434 Accuracy: 97.97% 23 Validation loss: 0.097838 Best loss: 0.055434 Accuracy: 97.81% 24 Validation loss: 0.071135 Best loss: 0.055434 Accuracy: 98.59% 25 Validation loss: 0.077825 Best loss: 0.055434 Accuracy: 98.48% 26 Validation loss: 0.107212 Best loss: 0.055434 Accuracy: 97.77% 27 Validation loss: 0.097347 Best loss: 0.055434 Accuracy: 98.32% 28 Validation loss: 0.099860 Best loss: 0.055434 Accuracy: 98.28% 29 Validation loss: 0.098857 Best loss: 0.055434 Accuracy: 98.36% 30 Validation loss: 0.082407 Best loss: 0.055434 Accuracy: 98.44% 31 Validation loss: 0.090734 Best loss: 0.055434 Accuracy: 98.32% 32 Validation loss: 0.298057 Best loss: 0.055434 Accuracy: 96.64% 33 Validation loss: 75965.546875 Best loss: 0.055434 Accuracy: 19.27% 34 Validation loss: 1591.949463 Best loss: 0.055434 Accuracy: 29.01% 35 Validation loss: 40.889915 Best loss: 0.055434 Accuracy: 85.22% 36 Validation loss: 18.931719 Best loss: 0.055434 Accuracy: 90.27% 37 Validation loss: 16.521729 Best loss: 0.055434 Accuracy: 89.80% 38 Validation loss: 12.614181 Best loss: 0.055434 Accuracy: 91.67% 39 Validation loss: 10.593999 Best loss: 0.055434 Accuracy: 91.99% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=50, batch_size=500, learning_rate=0.05, total= 5.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.102024 Best loss: 0.102024 Accuracy: 97.34% 1 Validation loss: 0.115835 Best loss: 0.102024 Accuracy: 97.15% 2 Validation loss: 0.148725 Best loss: 0.102024 Accuracy: 97.50% 3 Validation loss: 10.643847 Best loss: 0.102024 Accuracy: 88.12% 4 Validation loss: 0.928213 Best loss: 0.102024 Accuracy: 96.44% 5 Validation loss: 0.690542 Best loss: 0.102024 Accuracy: 95.70% 6 Validation loss: 0.812307 Best loss: 0.102024 Accuracy: 94.96% 7 Validation loss: 0.525574 Best loss: 0.102024 Accuracy: 96.29% 8 Validation loss: 0.722452 Best loss: 0.102024 Accuracy: 93.51% 9 Validation loss: 0.941825 Best loss: 0.102024 Accuracy: 92.18% 10 Validation loss: 0.407897 Best loss: 0.102024 Accuracy: 96.56% 11 Validation loss: 0.534575 Best loss: 0.102024 Accuracy: 96.68% 12 Validation loss: 0.368070 Best loss: 0.102024 Accuracy: 97.22% 13 Validation loss: 0.410520 Best loss: 0.102024 Accuracy: 95.27% 14 Validation loss: 0.414788 Best loss: 0.102024 Accuracy: 97.54% 15 Validation loss: 0.243067 Best loss: 0.102024 Accuracy: 97.54% 16 Validation loss: 0.377252 Best loss: 0.102024 Accuracy: 94.84% 17 Validation loss: 0.393079 Best loss: 0.102024 Accuracy: 97.30% 18 Validation loss: 0.337814 Best loss: 0.102024 Accuracy: 97.30% 19 Validation loss: 0.409032 Best loss: 0.102024 Accuracy: 97.69% 20 Validation loss: 0.228756 Best loss: 0.102024 Accuracy: 97.50% 21 Validation loss: 0.299810 Best loss: 0.102024 Accuracy: 97.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 11.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.125687 Best loss: 0.125687 Accuracy: 96.83% 1 Validation loss: 23.981316 Best loss: 0.125687 Accuracy: 75.18% 2 Validation loss: 0.251723 Best loss: 0.125687 Accuracy: 95.70% 3 Validation loss: 0.195717 Best loss: 0.125687 Accuracy: 96.36% 4 Validation loss: 0.152933 Best loss: 0.125687 Accuracy: 96.60% 5 Validation loss: 0.271237 Best loss: 0.125687 Accuracy: 94.88% 6 Validation loss: 0.166357 Best loss: 0.125687 Accuracy: 96.83% 7 Validation loss: 0.149770 Best loss: 0.125687 Accuracy: 96.91% 8 Validation loss: 0.201845 Best loss: 0.125687 Accuracy: 96.13% 9 Validation loss: 0.157877 Best loss: 0.125687 Accuracy: 96.91% 10 Validation loss: 0.219799 Best loss: 0.125687 Accuracy: 96.33% 11 Validation loss: 0.197705 Best loss: 0.125687 Accuracy: 97.11% 12 Validation loss: 0.236909 Best loss: 0.125687 Accuracy: 97.07% 13 Validation loss: 0.169058 Best loss: 0.125687 Accuracy: 97.22% 14 Validation loss: 0.190730 Best loss: 0.125687 Accuracy: 97.26% 15 Validation loss: 0.230055 Best loss: 0.125687 Accuracy: 97.38% 16 Validation loss: 0.227916 Best loss: 0.125687 Accuracy: 97.34% 17 Validation loss: 226.091751 Best loss: 0.125687 Accuracy: 92.14% 18 Validation loss: 13.008236 Best loss: 0.125687 Accuracy: 95.62% 19 Validation loss: 12.847130 Best loss: 0.125687 Accuracy: 95.74% 20 Validation loss: 5.821720 Best loss: 0.125687 Accuracy: 96.33% 21 Validation loss: 7.829120 Best loss: 0.125687 Accuracy: 95.15% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 11.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02 0 Validation loss: 0.087917 Best loss: 0.087917 Accuracy: 97.69% 1 Validation loss: 0.319121 Best loss: 0.087917 Accuracy: 96.56% 2 Validation loss: 0.435326 Best loss: 0.087917 Accuracy: 94.06% 3 Validation loss: 0.284446 Best loss: 0.087917 Accuracy: 96.60% 4 Validation loss: 0.183403 Best loss: 0.087917 Accuracy: 96.79% 5 Validation loss: 0.139043 Best loss: 0.087917 Accuracy: 96.99% 6 Validation loss: 0.116223 Best loss: 0.087917 Accuracy: 97.69% 7 Validation loss: 0.118389 Best loss: 0.087917 Accuracy: 98.01% 8 Validation loss: 0.173973 Best loss: 0.087917 Accuracy: 95.90% 9 Validation loss: 0.112023 Best loss: 0.087917 Accuracy: 98.16% 10 Validation loss: 0.132680 Best loss: 0.087917 Accuracy: 97.73% 11 Validation loss: 0.110227 Best loss: 0.087917 Accuracy: 98.24% 12 Validation loss: 0.123875 Best loss: 0.087917 Accuracy: 98.01% 13 Validation loss: 0.148492 Best loss: 0.087917 Accuracy: 97.58% 14 Validation loss: 0.129190 Best loss: 0.087917 Accuracy: 97.81% 15 Validation loss: 0.129665 Best loss: 0.087917 Accuracy: 97.77% 16 Validation loss: 0.120344 Best loss: 0.087917 Accuracy: 97.81% 17 Validation loss: 0.107339 Best loss: 0.087917 Accuracy: 97.89% 18 Validation loss: 0.346423 Best loss: 0.087917 Accuracy: 97.50% 19 Validation loss: 749.856995 Best loss: 0.087917 Accuracy: 64.11% 20 Validation loss: 7.881784 Best loss: 0.087917 Accuracy: 92.73% 21 Validation loss: 3.316260 Best loss: 0.087917 Accuracy: 96.13% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=70, batch_size=100, learning_rate=0.02, total= 11.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01 0 Validation loss: 0.729064 Best loss: 0.729064 Accuracy: 94.33% 1 Validation loss: 0.286473 Best loss: 0.286473 Accuracy: 96.25% 2 Validation loss: 499.945770 Best loss: 0.286473 Accuracy: 61.85% 3 Validation loss: 3.941091 Best loss: 0.286473 Accuracy: 89.29% 4 Validation loss: 2.132259 Best loss: 0.286473 Accuracy: 94.33% 5 Validation loss: 1.416036 Best loss: 0.286473 Accuracy: 96.64% 6 Validation loss: 85.976578 Best loss: 0.286473 Accuracy: 94.41% 7 Validation loss: 2.395885 Best loss: 0.286473 Accuracy: 96.44% 8 Validation loss: 264.726562 Best loss: 0.286473 Accuracy: 86.79% 9 Validation loss: 6.970783 Best loss: 0.286473 Accuracy: 93.04% 10 Validation loss: 22.561247 Best loss: 0.286473 Accuracy: 96.21% 11 Validation loss: 5.021589 Best loss: 0.286473 Accuracy: 93.47% 12 Validation loss: 3.802211 Best loss: 0.286473 Accuracy: 95.97% 13 Validation loss: 1.037142 Best loss: 0.286473 Accuracy: 97.65% 14 Validation loss: 11.088366 Best loss: 0.286473 Accuracy: 97.07% 15 Validation loss: 13.884163 Best loss: 0.286473 Accuracy: 96.95% 16 Validation loss: 15.182236 Best loss: 0.286473 Accuracy: 94.06% 17 Validation loss: 8.266023 Best loss: 0.286473 Accuracy: 95.70% 18 Validation loss: 19.708853 Best loss: 0.286473 Accuracy: 96.64% 19 Validation loss: 22.359173 Best loss: 0.286473 Accuracy: 94.53% 20 Validation loss: 51.983276 Best loss: 0.286473 Accuracy: 95.31% 21 Validation loss: 10.978294 Best loss: 0.286473 Accuracy: 97.77% 22 Validation loss: 62.488914 Best loss: 0.286473 Accuracy: 96.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01, total= 1.7min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01 0 Validation loss: 0.193438 Best loss: 0.193438 Accuracy: 94.84% 1 Validation loss: 4.068196 Best loss: 0.193438 Accuracy: 82.45% 2 Validation loss: 0.184150 Best loss: 0.184150 Accuracy: 95.82% 3 Validation loss: 281.588776 Best loss: 0.184150 Accuracy: 86.75% 4 Validation loss: 18.381681 Best loss: 0.184150 Accuracy: 96.21% 5 Validation loss: 2.887956 Best loss: 0.184150 Accuracy: 97.15% 6 Validation loss: 7.686343 Best loss: 0.184150 Accuracy: 96.33% 7 Validation loss: 11.538942 Best loss: 0.184150 Accuracy: 95.19% 8 Validation loss: 1.142919 Best loss: 0.184150 Accuracy: 97.22% 9 Validation loss: 105.652840 Best loss: 0.184150 Accuracy: 94.57% 10 Validation loss: 13.885515 Best loss: 0.184150 Accuracy: 97.03% 11 Validation loss: 5.906831 Best loss: 0.184150 Accuracy: 95.62% 12 Validation loss: 3.761411 Best loss: 0.184150 Accuracy: 94.33% 13 Validation loss: 79.576645 Best loss: 0.184150 Accuracy: 87.92% 14 Validation loss: 7.516062 Best loss: 0.184150 Accuracy: 97.93% 15 Validation loss: 172.281204 Best loss: 0.184150 Accuracy: 94.14% 16 Validation loss: 13.633722 Best loss: 0.184150 Accuracy: 93.75% 17 Validation loss: 11.941620 Best loss: 0.184150 Accuracy: 97.30% 18 Validation loss: 686.271790 Best loss: 0.184150 Accuracy: 96.05% 19 Validation loss: 47.542240 Best loss: 0.184150 Accuracy: 97.58% 20 Validation loss: 43.528790 Best loss: 0.184150 Accuracy: 97.19% 21 Validation loss: 22.044928 Best loss: 0.184150 Accuracy: 97.89% 22 Validation loss: 29.361326 Best loss: 0.184150 Accuracy: 97.58% 23 Validation loss: 26.499569 Best loss: 0.184150 Accuracy: 95.58% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01, total= 1.8min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01 0 Validation loss: 0.379055 Best loss: 0.379055 Accuracy: 93.98% 1 Validation loss: 0.293411 Best loss: 0.293411 Accuracy: 95.82% 2 Validation loss: 1.744731 Best loss: 0.293411 Accuracy: 94.92% 3 Validation loss: 2.583454 Best loss: 0.293411 Accuracy: 95.97% 4 Validation loss: 1.906730 Best loss: 0.293411 Accuracy: 96.29% 5 Validation loss: 0.717650 Best loss: 0.293411 Accuracy: 95.90% 6 Validation loss: 99.321487 Best loss: 0.293411 Accuracy: 93.78% 7 Validation loss: 8.261892 Best loss: 0.293411 Accuracy: 97.07% 8 Validation loss: 18.118559 Best loss: 0.293411 Accuracy: 94.96% 9 Validation loss: 5.961934 Best loss: 0.293411 Accuracy: 96.48% 10 Validation loss: 3.004120 Best loss: 0.293411 Accuracy: 97.19% 11 Validation loss: 14.309097 Best loss: 0.293411 Accuracy: 92.22% 12 Validation loss: 99.621490 Best loss: 0.293411 Accuracy: 75.10% 13 Validation loss: 69.748161 Best loss: 0.293411 Accuracy: 92.92% 14 Validation loss: 11.172760 Best loss: 0.293411 Accuracy: 96.76% 15 Validation loss: 22.243019 Best loss: 0.293411 Accuracy: 96.09% 16 Validation loss: 5.161487 Best loss: 0.293411 Accuracy: 97.38% 17 Validation loss: 61.199131 Best loss: 0.293411 Accuracy: 83.93% 18 Validation loss: 34.983425 Best loss: 0.293411 Accuracy: 97.65% 19 Validation loss: 39.887814 Best loss: 0.293411 Accuracy: 96.76% 20 Validation loss: 41.164520 Best loss: 0.293411 Accuracy: 97.38% 21 Validation loss: 25.366238 Best loss: 0.293411 Accuracy: 97.38% 22 Validation loss: 16.120886 Best loss: 0.293411 Accuracy: 97.42% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=10, learning_rate=0.01, total= 1.7min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1 0 Validation loss: 850.403625 Best loss: 850.403625 Accuracy: 90.85% 1 Validation loss: 336.769684 Best loss: 336.769684 Accuracy: 93.75% 2 Validation loss: 346.975128 Best loss: 336.769684 Accuracy: 92.61% 3 Validation loss: 134.534592 Best loss: 134.534592 Accuracy: 93.82% 4 Validation loss: 139.472549 Best loss: 134.534592 Accuracy: 94.92% 5 Validation loss: 182.103729 Best loss: 134.534592 Accuracy: 91.95% 6 Validation loss: 125.848198 Best loss: 125.848198 Accuracy: 94.06% 7 Validation loss: 81.838829 Best loss: 81.838829 Accuracy: 95.93% 8 Validation loss: 82.043762 Best loss: 81.838829 Accuracy: 95.82% 9 Validation loss: 97.803688 Best loss: 81.838829 Accuracy: 95.50% 10 Validation loss: 94.638062 Best loss: 81.838829 Accuracy: 94.02% 11 Validation loss: 131.307220 Best loss: 81.838829 Accuracy: 92.49% 12 Validation loss: 42.942501 Best loss: 42.942501 Accuracy: 96.95% 13 Validation loss: 40.969852 Best loss: 40.969852 Accuracy: 96.01% 14 Validation loss: 33.598263 Best loss: 33.598263 Accuracy: 96.48% 15 Validation loss: 61.034412 Best loss: 33.598263 Accuracy: 95.00% 16 Validation loss: 59.101837 Best loss: 33.598263 Accuracy: 94.18% 17 Validation loss: 50.972473 Best loss: 33.598263 Accuracy: 95.00% 18 Validation loss: 37.112999 Best loss: 33.598263 Accuracy: 96.87% 19 Validation loss: 24.283026 Best loss: 24.283026 Accuracy: 97.30% 20 Validation loss: 29.697607 Best loss: 24.283026 Accuracy: 96.40% 21 Validation loss: 29.802830 Best loss: 24.283026 Accuracy: 96.87% 22 Validation loss: 18.582729 Best loss: 18.582729 Accuracy: 97.73% 23 Validation loss: 25.016119 Best loss: 18.582729 Accuracy: 96.29% 24 Validation loss: 54.633755 Best loss: 18.582729 Accuracy: 91.20% 25 Validation loss: 14.380316 Best loss: 14.380316 Accuracy: 97.50% 26 Validation loss: 32.612667 Best loss: 14.380316 Accuracy: 97.11% 27 Validation loss: 47.524761 Best loss: 14.380316 Accuracy: 96.01% 28 Validation loss: 16.379150 Best loss: 14.380316 Accuracy: 97.54% 29 Validation loss: 18.829224 Best loss: 14.380316 Accuracy: 97.54% 30 Validation loss: 18.693233 Best loss: 14.380316 Accuracy: 96.21% 31 Validation loss: 31.111309 Best loss: 14.380316 Accuracy: 96.52% 32 Validation loss: 6410286.500000 Best loss: 14.380316 Accuracy: 77.44% 33 Validation loss: 1570077.875000 Best loss: 14.380316 Accuracy: 88.58% 34 Validation loss: 939800.625000 Best loss: 14.380316 Accuracy: 90.23% 35 Validation loss: 593135.562500 Best loss: 14.380316 Accuracy: 92.22% 36 Validation loss: 1036682.187500 Best loss: 14.380316 Accuracy: 83.50% 37 Validation loss: 843884.687500 Best loss: 14.380316 Accuracy: 90.97% 38 Validation loss: 510979.750000 Best loss: 14.380316 Accuracy: 88.31% 39 Validation loss: 816683.125000 Best loss: 14.380316 Accuracy: 84.52% 40 Validation loss: 684629.687500 Best loss: 14.380316 Accuracy: 86.43% 41 Validation loss: 323395.937500 Best loss: 14.380316 Accuracy: 95.27% 42 Validation loss: 739403.375000 Best loss: 14.380316 Accuracy: 86.43% 43 Validation loss: 547789.187500 Best loss: 14.380316 Accuracy: 90.30% 44 Validation loss: 343060.281250 Best loss: 14.380316 Accuracy: 95.11% 45 Validation loss: 383926.437500 Best loss: 14.380316 Accuracy: 95.35% 46 Validation loss: 443589.843750 Best loss: 14.380316 Accuracy: 93.82% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1, total= 23.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1 0 Validation loss: 33.464554 Best loss: 33.464554 Accuracy: 90.30% 1 Validation loss: 25.523787 Best loss: 25.523787 Accuracy: 91.20% 2 Validation loss: 15.509095 Best loss: 15.509095 Accuracy: 90.42% 3 Validation loss: 31.631643 Best loss: 15.509095 Accuracy: 89.91% 4 Validation loss: 7.290944 Best loss: 7.290944 Accuracy: 95.23% 5 Validation loss: 12.959599 Best loss: 7.290944 Accuracy: 91.32% 6 Validation loss: 3.368996 Best loss: 3.368996 Accuracy: 96.33% 7 Validation loss: 2.038879 Best loss: 2.038879 Accuracy: 96.25% 8 Validation loss: 2.979335 Best loss: 2.038879 Accuracy: 95.58% 9 Validation loss: 2.955154 Best loss: 2.038879 Accuracy: 96.68% 10 Validation loss: 2.744008 Best loss: 2.038879 Accuracy: 96.01% 11 Validation loss: 4.531036 Best loss: 2.038879 Accuracy: 95.62% 12 Validation loss: 115945944.000000 Best loss: 2.038879 Accuracy: 19.08% 13 Validation loss: 1306821.375000 Best loss: 2.038879 Accuracy: 79.91% 14 Validation loss: 494617.906250 Best loss: 2.038879 Accuracy: 85.93% 15 Validation loss: 333539.625000 Best loss: 2.038879 Accuracy: 88.74% 16 Validation loss: 196147.359375 Best loss: 2.038879 Accuracy: 92.49% 17 Validation loss: 180137.843750 Best loss: 2.038879 Accuracy: 91.95% 18 Validation loss: 181102.796875 Best loss: 2.038879 Accuracy: 94.37% 19 Validation loss: 194977.015625 Best loss: 2.038879 Accuracy: 93.43% 20 Validation loss: 142183.250000 Best loss: 2.038879 Accuracy: 94.61% 21 Validation loss: 158233.062500 Best loss: 2.038879 Accuracy: 93.90% 22 Validation loss: 175466.171875 Best loss: 2.038879 Accuracy: 93.63% 23 Validation loss: 146803.234375 Best loss: 2.038879 Accuracy: 91.56% 24 Validation loss: 163517.015625 Best loss: 2.038879 Accuracy: 93.24% 25 Validation loss: 87455.031250 Best loss: 2.038879 Accuracy: 95.07% 26 Validation loss: 136608.781250 Best loss: 2.038879 Accuracy: 95.15% 27 Validation loss: 73771.000000 Best loss: 2.038879 Accuracy: 96.52% 28 Validation loss: 152589.921875 Best loss: 2.038879 Accuracy: 93.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1, total= 15.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1 0 Validation loss: 34.428257 Best loss: 34.428257 Accuracy: 91.20% 1 Validation loss: 20.637756 Best loss: 20.637756 Accuracy: 91.48% 2 Validation loss: 76.767677 Best loss: 20.637756 Accuracy: 90.81% 3 Validation loss: 7127374.000000 Best loss: 20.637756 Accuracy: 47.30% 4 Validation loss: 190078.265625 Best loss: 20.637756 Accuracy: 82.29% 5 Validation loss: 79186.023438 Best loss: 20.637756 Accuracy: 88.47% 6 Validation loss: 143436.140625 Best loss: 20.637756 Accuracy: 79.87% 7 Validation loss: 25435.634766 Best loss: 20.637756 Accuracy: 93.43% 8 Validation loss: 27265.447266 Best loss: 20.637756 Accuracy: 91.91% 9 Validation loss: 55375.625000 Best loss: 20.637756 Accuracy: 86.75% 10 Validation loss: 39503.312500 Best loss: 20.637756 Accuracy: 92.77% 11 Validation loss: 30446.621094 Best loss: 20.637756 Accuracy: 92.89% 12 Validation loss: 31261.285156 Best loss: 20.637756 Accuracy: 92.06% 13 Validation loss: 46909.523438 Best loss: 20.637756 Accuracy: 90.81% 14 Validation loss: 22680.373047 Best loss: 20.637756 Accuracy: 92.03% 15 Validation loss: 18670.757812 Best loss: 20.637756 Accuracy: 95.62% 16 Validation loss: 15371.139648 Best loss: 20.637756 Accuracy: 95.07% 17 Validation loss: 7296.317383 Best loss: 20.637756 Accuracy: 96.83% 18 Validation loss: 12917.583984 Best loss: 20.637756 Accuracy: 96.60% 19 Validation loss: 34145.214844 Best loss: 20.637756 Accuracy: 93.86% 20 Validation loss: 35010.746094 Best loss: 20.637756 Accuracy: 91.91% 21 Validation loss: 23914.339844 Best loss: 20.637756 Accuracy: 92.49% 22 Validation loss: 18856.744141 Best loss: 20.637756 Accuracy: 95.62% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=160, batch_size=100, learning_rate=0.1, total= 11.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02 0 Validation loss: 0.104175 Best loss: 0.104175 Accuracy: 97.38% 1 Validation loss: 0.117586 Best loss: 0.104175 Accuracy: 97.54% 2 Validation loss: 0.108333 Best loss: 0.104175 Accuracy: 97.34% 3 Validation loss: 0.122912 Best loss: 0.104175 Accuracy: 96.21% 4 Validation loss: 0.102227 Best loss: 0.102227 Accuracy: 98.01% 5 Validation loss: 0.094710 Best loss: 0.094710 Accuracy: 98.20% 6 Validation loss: 0.099074 Best loss: 0.094710 Accuracy: 98.08% 7 Validation loss: 0.342326 Best loss: 0.094710 Accuracy: 92.30% 8 Validation loss: 0.376116 Best loss: 0.094710 Accuracy: 85.38% 9 Validation loss: 0.227018 Best loss: 0.094710 Accuracy: 94.72% 10 Validation loss: 0.162873 Best loss: 0.094710 Accuracy: 96.13% 11 Validation loss: 0.140025 Best loss: 0.094710 Accuracy: 97.03% 12 Validation loss: 0.172240 Best loss: 0.094710 Accuracy: 96.87% 13 Validation loss: 0.160971 Best loss: 0.094710 Accuracy: 96.56% 14 Validation loss: 0.182086 Best loss: 0.094710 Accuracy: 96.21% 15 Validation loss: 0.230638 Best loss: 0.094710 Accuracy: 95.74% 16 Validation loss: 0.565395 Best loss: 0.094710 Accuracy: 74.78% 17 Validation loss: 0.481577 Best loss: 0.094710 Accuracy: 78.93% 18 Validation loss: 0.527615 Best loss: 0.094710 Accuracy: 75.53% 19 Validation loss: 0.456358 Best loss: 0.094710 Accuracy: 76.90% 20 Validation loss: 0.461231 Best loss: 0.094710 Accuracy: 79.40% 21 Validation loss: 0.507518 Best loss: 0.094710 Accuracy: 75.25% 22 Validation loss: 0.527901 Best loss: 0.094710 Accuracy: 76.90% 23 Validation loss: 0.443119 Best loss: 0.094710 Accuracy: 78.42% 24 Validation loss: 0.489825 Best loss: 0.094710 Accuracy: 76.90% 25 Validation loss: 0.448147 Best loss: 0.094710 Accuracy: 75.14% 26 Validation loss: 0.433520 Best loss: 0.094710 Accuracy: 79.20% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02, total= 11.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02 0 Validation loss: 0.100262 Best loss: 0.100262 Accuracy: 96.87% 1 Validation loss: 0.130332 Best loss: 0.100262 Accuracy: 96.33% 2 Validation loss: 0.118281 Best loss: 0.100262 Accuracy: 97.26% 3 Validation loss: 0.064055 Best loss: 0.064055 Accuracy: 98.63% 4 Validation loss: 0.204077 Best loss: 0.064055 Accuracy: 96.17% 5 Validation loss: 0.534768 Best loss: 0.064055 Accuracy: 75.61% 6 Validation loss: 0.163318 Best loss: 0.064055 Accuracy: 95.90% 7 Validation loss: 0.148038 Best loss: 0.064055 Accuracy: 96.29% 8 Validation loss: 0.123454 Best loss: 0.064055 Accuracy: 97.54% 9 Validation loss: 0.122845 Best loss: 0.064055 Accuracy: 97.50% 10 Validation loss: 0.173010 Best loss: 0.064055 Accuracy: 95.82% 11 Validation loss: 0.099025 Best loss: 0.064055 Accuracy: 97.15% 12 Validation loss: 0.143184 Best loss: 0.064055 Accuracy: 97.30% 13 Validation loss: 0.095946 Best loss: 0.064055 Accuracy: 97.85% 14 Validation loss: 0.155078 Best loss: 0.064055 Accuracy: 96.79% 15 Validation loss: 0.116736 Best loss: 0.064055 Accuracy: 97.34% 16 Validation loss: 0.117428 Best loss: 0.064055 Accuracy: 98.12% 17 Validation loss: 0.158950 Best loss: 0.064055 Accuracy: 97.54% 18 Validation loss: 0.081340 Best loss: 0.064055 Accuracy: 98.16% 19 Validation loss: 0.128592 Best loss: 0.064055 Accuracy: 97.54% 20 Validation loss: 0.120003 Best loss: 0.064055 Accuracy: 97.42% 21 Validation loss: 0.129002 Best loss: 0.064055 Accuracy: 97.81% 22 Validation loss: 0.129184 Best loss: 0.064055 Accuracy: 97.54% 23 Validation loss: 0.216669 Best loss: 0.064055 Accuracy: 95.35% 24 Validation loss: 0.167843 Best loss: 0.064055 Accuracy: 97.93% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02, total= 10.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02 0 Validation loss: 0.107893 Best loss: 0.107893 Accuracy: 97.22% 1 Validation loss: 0.096697 Best loss: 0.096697 Accuracy: 97.38% 2 Validation loss: 0.255332 Best loss: 0.096697 Accuracy: 96.05% 3 Validation loss: 0.347833 Best loss: 0.096697 Accuracy: 93.39% 4 Validation loss: 0.184912 Best loss: 0.096697 Accuracy: 95.97% 5 Validation loss: 0.117708 Best loss: 0.096697 Accuracy: 97.26% 6 Validation loss: 0.099796 Best loss: 0.096697 Accuracy: 97.65% 7 Validation loss: 0.109082 Best loss: 0.096697 Accuracy: 97.93% 8 Validation loss: 0.106203 Best loss: 0.096697 Accuracy: 97.89% 9 Validation loss: 0.107012 Best loss: 0.096697 Accuracy: 98.32% 10 Validation loss: 0.072560 Best loss: 0.072560 Accuracy: 98.28% 11 Validation loss: 0.122552 Best loss: 0.072560 Accuracy: 97.77% 12 Validation loss: 0.131580 Best loss: 0.072560 Accuracy: 97.69% 13 Validation loss: 1.634258 Best loss: 0.072560 Accuracy: 20.91% 14 Validation loss: 1.622727 Best loss: 0.072560 Accuracy: 19.08% 15 Validation loss: 1.630534 Best loss: 0.072560 Accuracy: 19.08% 16 Validation loss: 1.675099 Best loss: 0.072560 Accuracy: 18.73% 17 Validation loss: 1.624748 Best loss: 0.072560 Accuracy: 19.08% 18 Validation loss: 1.641751 Best loss: 0.072560 Accuracy: 22.01% 19 Validation loss: 1.638702 Best loss: 0.072560 Accuracy: 20.91% 20 Validation loss: 1.661677 Best loss: 0.072560 Accuracy: 19.27% 21 Validation loss: 1.669129 Best loss: 0.072560 Accuracy: 19.27% 22 Validation loss: 1.625988 Best loss: 0.072560 Accuracy: 22.01% 23 Validation loss: 1.624628 Best loss: 0.072560 Accuracy: 22.01% 24 Validation loss: 1.631172 Best loss: 0.072560 Accuracy: 22.01% 25 Validation loss: 1.617089 Best loss: 0.072560 Accuracy: 20.91% 26 Validation loss: 1.623339 Best loss: 0.072560 Accuracy: 19.08% 27 Validation loss: 1.639719 Best loss: 0.072560 Accuracy: 18.73% 28 Validation loss: 1.613069 Best loss: 0.072560 Accuracy: 22.01% 29 Validation loss: 1.632918 Best loss: 0.072560 Accuracy: 18.73% 30 Validation loss: 1.624189 Best loss: 0.072560 Accuracy: 19.27% 31 Validation loss: 1.626345 Best loss: 0.072560 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=100, learning_rate=0.02, total= 13.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02 0 Validation loss: 0.197263 Best loss: 0.197263 Accuracy: 95.39% 1 Validation loss: 0.136613 Best loss: 0.136613 Accuracy: 96.36% 2 Validation loss: 0.122228 Best loss: 0.122228 Accuracy: 96.40% 3 Validation loss: 0.110332 Best loss: 0.110332 Accuracy: 96.33% 4 Validation loss: 0.118622 Best loss: 0.110332 Accuracy: 96.99% 5 Validation loss: 0.118651 Best loss: 0.110332 Accuracy: 96.68% 6 Validation loss: 0.161918 Best loss: 0.110332 Accuracy: 95.74% 7 Validation loss: 0.132039 Best loss: 0.110332 Accuracy: 95.78% 8 Validation loss: 0.118300 Best loss: 0.110332 Accuracy: 96.68% 9 Validation loss: 0.125507 Best loss: 0.110332 Accuracy: 96.29% 10 Validation loss: 0.115947 Best loss: 0.110332 Accuracy: 96.87% 11 Validation loss: 0.116463 Best loss: 0.110332 Accuracy: 96.40% 12 Validation loss: 0.132445 Best loss: 0.110332 Accuracy: 96.87% 13 Validation loss: 0.134595 Best loss: 0.110332 Accuracy: 96.76% 14 Validation loss: 0.130330 Best loss: 0.110332 Accuracy: 96.64% 15 Validation loss: 0.126696 Best loss: 0.110332 Accuracy: 96.72% 16 Validation loss: 0.116546 Best loss: 0.110332 Accuracy: 96.99% 17 Validation loss: 0.131190 Best loss: 0.110332 Accuracy: 97.03% 18 Validation loss: 0.137345 Best loss: 0.110332 Accuracy: 96.64% 19 Validation loss: 0.130982 Best loss: 0.110332 Accuracy: 95.74% 20 Validation loss: 0.129079 Best loss: 0.110332 Accuracy: 96.91% 21 Validation loss: 0.151698 Best loss: 0.110332 Accuracy: 96.29% 22 Validation loss: 0.150095 Best loss: 0.110332 Accuracy: 96.13% 23 Validation loss: 0.136464 Best loss: 0.110332 Accuracy: 96.95% 24 Validation loss: 0.117865 Best loss: 0.110332 Accuracy: 96.79% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02, total= 10.6s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02 0 Validation loss: 0.155509 Best loss: 0.155509 Accuracy: 96.09% 1 Validation loss: 0.124830 Best loss: 0.124830 Accuracy: 96.33% 2 Validation loss: 0.119341 Best loss: 0.119341 Accuracy: 96.44% 3 Validation loss: 0.133911 Best loss: 0.119341 Accuracy: 96.09% 4 Validation loss: 0.118551 Best loss: 0.118551 Accuracy: 96.76% 5 Validation loss: 0.123160 Best loss: 0.118551 Accuracy: 96.09% 6 Validation loss: 0.103847 Best loss: 0.103847 Accuracy: 97.07% 7 Validation loss: 0.100502 Best loss: 0.100502 Accuracy: 97.11% 8 Validation loss: 0.104134 Best loss: 0.100502 Accuracy: 97.34% 9 Validation loss: 0.104285 Best loss: 0.100502 Accuracy: 97.07% 10 Validation loss: 0.117892 Best loss: 0.100502 Accuracy: 96.72% 11 Validation loss: 0.118585 Best loss: 0.100502 Accuracy: 96.68% 12 Validation loss: 0.106598 Best loss: 0.100502 Accuracy: 97.15% 13 Validation loss: 0.134617 Best loss: 0.100502 Accuracy: 96.60% 14 Validation loss: 0.132133 Best loss: 0.100502 Accuracy: 96.91% 15 Validation loss: 0.111290 Best loss: 0.100502 Accuracy: 97.07% 16 Validation loss: 0.142708 Best loss: 0.100502 Accuracy: 96.64% 17 Validation loss: 0.123269 Best loss: 0.100502 Accuracy: 97.03% 18 Validation loss: 0.123326 Best loss: 0.100502 Accuracy: 96.56% 19 Validation loss: 0.124168 Best loss: 0.100502 Accuracy: 96.91% 20 Validation loss: 0.116776 Best loss: 0.100502 Accuracy: 96.87% 21 Validation loss: 0.117025 Best loss: 0.100502 Accuracy: 96.56% 22 Validation loss: 0.140092 Best loss: 0.100502 Accuracy: 96.56% 23 Validation loss: 0.135246 Best loss: 0.100502 Accuracy: 96.99% 24 Validation loss: 0.145748 Best loss: 0.100502 Accuracy: 96.36% 25 Validation loss: 0.143924 Best loss: 0.100502 Accuracy: 96.87% 26 Validation loss: 0.138756 Best loss: 0.100502 Accuracy: 96.87% 27 Validation loss: 0.156632 Best loss: 0.100502 Accuracy: 96.64% 28 Validation loss: 0.149727 Best loss: 0.100502 Accuracy: 96.64% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02, total= 12.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02 0 Validation loss: 0.160756 Best loss: 0.160756 Accuracy: 95.74% 1 Validation loss: 0.136592 Best loss: 0.136592 Accuracy: 96.76% 2 Validation loss: 0.130938 Best loss: 0.130938 Accuracy: 96.68% 3 Validation loss: 0.111893 Best loss: 0.111893 Accuracy: 96.99% 4 Validation loss: 0.104256 Best loss: 0.104256 Accuracy: 97.38% 5 Validation loss: 0.110129 Best loss: 0.104256 Accuracy: 97.26% 6 Validation loss: 0.097090 Best loss: 0.097090 Accuracy: 97.19% 7 Validation loss: 0.145146 Best loss: 0.097090 Accuracy: 96.44% 8 Validation loss: 0.093107 Best loss: 0.093107 Accuracy: 97.62% 9 Validation loss: 0.107980 Best loss: 0.093107 Accuracy: 97.19% 10 Validation loss: 0.100624 Best loss: 0.093107 Accuracy: 97.58% 11 Validation loss: 0.108981 Best loss: 0.093107 Accuracy: 97.34% 12 Validation loss: 0.103289 Best loss: 0.093107 Accuracy: 97.22% 13 Validation loss: 0.092855 Best loss: 0.092855 Accuracy: 97.62% 14 Validation loss: 0.098182 Best loss: 0.092855 Accuracy: 97.58% 15 Validation loss: 0.091933 Best loss: 0.091933 Accuracy: 97.46% 16 Validation loss: 0.095014 Best loss: 0.091933 Accuracy: 97.58% 17 Validation loss: 0.098227 Best loss: 0.091933 Accuracy: 97.34% 18 Validation loss: 0.131417 Best loss: 0.091933 Accuracy: 96.72% 19 Validation loss: 0.101071 Best loss: 0.091933 Accuracy: 97.46% 20 Validation loss: 0.109709 Best loss: 0.091933 Accuracy: 97.30% 21 Validation loss: 0.101462 Best loss: 0.091933 Accuracy: 97.65% 22 Validation loss: 0.094337 Best loss: 0.091933 Accuracy: 97.50% 23 Validation loss: 0.104698 Best loss: 0.091933 Accuracy: 97.42% 24 Validation loss: 0.112113 Best loss: 0.091933 Accuracy: 97.42% 25 Validation loss: 0.104471 Best loss: 0.091933 Accuracy: 97.34% 26 Validation loss: 0.101661 Best loss: 0.091933 Accuracy: 97.38% 27 Validation loss: 0.111230 Best loss: 0.091933 Accuracy: 96.99% 28 Validation loss: 0.091289 Best loss: 0.091289 Accuracy: 97.81% 29 Validation loss: 0.095398 Best loss: 0.091289 Accuracy: 97.77% 30 Validation loss: 0.120339 Best loss: 0.091289 Accuracy: 97.54% 31 Validation loss: 0.104513 Best loss: 0.091289 Accuracy: 97.50% 32 Validation loss: 0.102787 Best loss: 0.091289 Accuracy: 97.69% 33 Validation loss: 0.098384 Best loss: 0.091289 Accuracy: 97.77% 34 Validation loss: 0.335006 Best loss: 0.091289 Accuracy: 97.77% 35 Validation loss: 0.100602 Best loss: 0.091289 Accuracy: 97.73% 36 Validation loss: 0.120759 Best loss: 0.091289 Accuracy: 97.03% 37 Validation loss: 0.105321 Best loss: 0.091289 Accuracy: 97.85% 38 Validation loss: 0.129511 Best loss: 0.091289 Accuracy: 97.54% 39 Validation loss: 0.115880 Best loss: 0.091289 Accuracy: 97.38% 40 Validation loss: 0.102763 Best loss: 0.091289 Accuracy: 97.69% 41 Validation loss: 0.101381 Best loss: 0.091289 Accuracy: 97.77% 42 Validation loss: 0.110730 Best loss: 0.091289 Accuracy: 97.62% 43 Validation loss: 0.107626 Best loss: 0.091289 Accuracy: 97.85% 44 Validation loss: 0.109769 Best loss: 0.091289 Accuracy: 97.93% 45 Validation loss: 0.098356 Best loss: 0.091289 Accuracy: 97.73% 46 Validation loss: 0.102387 Best loss: 0.091289 Accuracy: 97.85% 47 Validation loss: 0.113617 Best loss: 0.091289 Accuracy: 97.93% 48 Validation loss: 0.119486 Best loss: 0.091289 Accuracy: 98.08% 49 Validation loss: 0.132154 Best loss: 0.091289 Accuracy: 97.97% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=10, batch_size=100, learning_rate=0.02, total= 20.7s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02 0 Validation loss: 0.175154 Best loss: 0.175154 Accuracy: 94.21% 1 Validation loss: 0.408192 Best loss: 0.175154 Accuracy: 91.13% 2 Validation loss: 0.198909 Best loss: 0.175154 Accuracy: 95.23% 3 Validation loss: 0.213392 Best loss: 0.175154 Accuracy: 93.63% 4 Validation loss: 0.189125 Best loss: 0.175154 Accuracy: 95.04% 5 Validation loss: 0.283010 Best loss: 0.175154 Accuracy: 94.18% 6 Validation loss: 0.302679 Best loss: 0.175154 Accuracy: 90.93% 7 Validation loss: 0.650330 Best loss: 0.175154 Accuracy: 75.02% 8 Validation loss: 0.413767 Best loss: 0.175154 Accuracy: 78.77% 9 Validation loss: 0.789069 Best loss: 0.175154 Accuracy: 58.01% 10 Validation loss: 0.748380 Best loss: 0.175154 Accuracy: 61.18% 11 Validation loss: 1.242519 Best loss: 0.175154 Accuracy: 38.27% 12 Validation loss: 1.206584 Best loss: 0.175154 Accuracy: 38.12% 13 Validation loss: 1.071106 Best loss: 0.175154 Accuracy: 40.50% 14 Validation loss: 1.123444 Best loss: 0.175154 Accuracy: 40.58% 15 Validation loss: 1.200557 Best loss: 0.175154 Accuracy: 40.58% 16 Validation loss: 1.399900 Best loss: 0.175154 Accuracy: 32.84% 17 Validation loss: 1.338760 Best loss: 0.175154 Accuracy: 41.20% 18 Validation loss: 1.229676 Best loss: 0.175154 Accuracy: 40.50% 19 Validation loss: 1.183276 Best loss: 0.175154 Accuracy: 42.53% 20 Validation loss: 1.184143 Best loss: 0.175154 Accuracy: 42.53% 21 Validation loss: 1.191341 Best loss: 0.175154 Accuracy: 40.38% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02, total= 1.3min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02 0 Validation loss: 0.186749 Best loss: 0.186749 Accuracy: 95.35% 1 Validation loss: 0.208389 Best loss: 0.186749 Accuracy: 94.96% 2 Validation loss: 0.162491 Best loss: 0.162491 Accuracy: 94.92% 3 Validation loss: 0.175371 Best loss: 0.162491 Accuracy: 94.76% 4 Validation loss: 0.160922 Best loss: 0.160922 Accuracy: 96.05% 5 Validation loss: 0.156023 Best loss: 0.156023 Accuracy: 95.62% 6 Validation loss: 0.577734 Best loss: 0.156023 Accuracy: 77.17% 7 Validation loss: 1.249876 Best loss: 0.156023 Accuracy: 77.95% 8 Validation loss: 1.282245 Best loss: 0.156023 Accuracy: 36.12% 9 Validation loss: 1.180313 Best loss: 0.156023 Accuracy: 42.38% 10 Validation loss: 1.204774 Best loss: 0.156023 Accuracy: 39.87% 11 Validation loss: 1.182780 Best loss: 0.156023 Accuracy: 39.68% 12 Validation loss: 1.187500 Best loss: 0.156023 Accuracy: 39.68% 13 Validation loss: 1.181499 Best loss: 0.156023 Accuracy: 39.29% 14 Validation loss: 1.181296 Best loss: 0.156023 Accuracy: 39.52% 15 Validation loss: 1.177513 Best loss: 0.156023 Accuracy: 41.20% 16 Validation loss: 1.186354 Best loss: 0.156023 Accuracy: 39.68% 17 Validation loss: 1.182037 Best loss: 0.156023 Accuracy: 39.29% 18 Validation loss: 1.183748 Best loss: 0.156023 Accuracy: 39.68% 19 Validation loss: 1.178820 Best loss: 0.156023 Accuracy: 39.68% 20 Validation loss: 1.179096 Best loss: 0.156023 Accuracy: 39.68% 21 Validation loss: 1.198198 Best loss: 0.156023 Accuracy: 39.29% 22 Validation loss: 1.196864 Best loss: 0.156023 Accuracy: 39.68% 23 Validation loss: 1.206793 Best loss: 0.156023 Accuracy: 39.68% 24 Validation loss: 1.193773 Best loss: 0.156023 Accuracy: 39.29% 25 Validation loss: 1.179078 Best loss: 0.156023 Accuracy: 41.20% 26 Validation loss: 1.178817 Best loss: 0.156023 Accuracy: 39.68% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02, total= 1.5min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02 0 Validation loss: 0.205903 Best loss: 0.205903 Accuracy: 94.25% 1 Validation loss: 0.122735 Best loss: 0.122735 Accuracy: 97.03% 2 Validation loss: 0.149516 Best loss: 0.122735 Accuracy: 96.29% 3 Validation loss: 0.212521 Best loss: 0.122735 Accuracy: 96.48% 4 Validation loss: 0.242549 Best loss: 0.122735 Accuracy: 94.21% 5 Validation loss: 0.330887 Best loss: 0.122735 Accuracy: 91.67% 6 Validation loss: 0.306163 Best loss: 0.122735 Accuracy: 95.82% 7 Validation loss: 0.252378 Best loss: 0.122735 Accuracy: 94.88% 8 Validation loss: 0.231685 Best loss: 0.122735 Accuracy: 95.82% 9 Validation loss: 0.664854 Best loss: 0.122735 Accuracy: 70.88% 10 Validation loss: 0.497880 Best loss: 0.122735 Accuracy: 77.95% 11 Validation loss: 0.291596 Best loss: 0.122735 Accuracy: 94.72% 12 Validation loss: 0.226452 Best loss: 0.122735 Accuracy: 95.11% 13 Validation loss: 0.274685 Best loss: 0.122735 Accuracy: 93.00% 14 Validation loss: 0.353443 Best loss: 0.122735 Accuracy: 95.27% 15 Validation loss: 0.812656 Best loss: 0.122735 Accuracy: 57.97% 16 Validation loss: 0.847268 Best loss: 0.122735 Accuracy: 59.34% 17 Validation loss: 1.191011 Best loss: 0.122735 Accuracy: 37.02% 18 Validation loss: 1.184591 Best loss: 0.122735 Accuracy: 40.15% 19 Validation loss: 1.190056 Best loss: 0.122735 Accuracy: 37.02% 20 Validation loss: 1.188479 Best loss: 0.122735 Accuracy: 37.41% 21 Validation loss: 1.212461 Best loss: 0.122735 Accuracy: 40.58% 22 Validation loss: 1.224920 Best loss: 0.122735 Accuracy: 40.62% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=30, batch_size=10, learning_rate=0.02, total= 1.4min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01 0 Validation loss: 0.103824 Best loss: 0.103824 Accuracy: 97.22% 1 Validation loss: 0.084765 Best loss: 0.084765 Accuracy: 97.93% 2 Validation loss: 0.069354 Best loss: 0.069354 Accuracy: 98.28% 3 Validation loss: 0.083534 Best loss: 0.069354 Accuracy: 98.01% 4 Validation loss: 0.078348 Best loss: 0.069354 Accuracy: 98.01% 5 Validation loss: 0.064682 Best loss: 0.064682 Accuracy: 98.44% 6 Validation loss: 0.069909 Best loss: 0.064682 Accuracy: 98.75% 7 Validation loss: 0.079489 Best loss: 0.064682 Accuracy: 98.44% 8 Validation loss: 0.083839 Best loss: 0.064682 Accuracy: 98.16% 9 Validation loss: 0.098391 Best loss: 0.064682 Accuracy: 98.44% 10 Validation loss: 0.088385 Best loss: 0.064682 Accuracy: 98.59% 11 Validation loss: 0.098388 Best loss: 0.064682 Accuracy: 98.67% 12 Validation loss: 0.064155 Best loss: 0.064155 Accuracy: 98.51% 13 Validation loss: 0.078880 Best loss: 0.064155 Accuracy: 98.83% 14 Validation loss: 0.122993 Best loss: 0.064155 Accuracy: 98.28% 15 Validation loss: 0.267132 Best loss: 0.064155 Accuracy: 96.91% 16 Validation loss: 0.081967 Best loss: 0.064155 Accuracy: 98.40% 17 Validation loss: 0.100666 Best loss: 0.064155 Accuracy: 98.48% 18 Validation loss: 0.123592 Best loss: 0.064155 Accuracy: 96.13% 19 Validation loss: 0.081863 Best loss: 0.064155 Accuracy: 98.36% 20 Validation loss: 0.150217 Best loss: 0.064155 Accuracy: 98.36% 21 Validation loss: 0.243342 Best loss: 0.064155 Accuracy: 98.44% 22 Validation loss: 0.265774 Best loss: 0.064155 Accuracy: 98.01% 23 Validation loss: 0.174954 Best loss: 0.064155 Accuracy: 98.44% 24 Validation loss: 0.204148 Best loss: 0.064155 Accuracy: 97.77% 25 Validation loss: 0.164853 Best loss: 0.064155 Accuracy: 97.97% 26 Validation loss: 0.153206 Best loss: 0.064155 Accuracy: 95.66% 27 Validation loss: 0.152940 Best loss: 0.064155 Accuracy: 96.99% 28 Validation loss: 0.231583 Best loss: 0.064155 Accuracy: 98.55% 29 Validation loss: 0.100253 Best loss: 0.064155 Accuracy: 98.87% 30 Validation loss: 0.156060 Best loss: 0.064155 Accuracy: 98.48% 31 Validation loss: 0.080692 Best loss: 0.064155 Accuracy: 98.75% 32 Validation loss: 0.193663 Best loss: 0.064155 Accuracy: 98.59% 33 Validation loss: 0.102300 Best loss: 0.064155 Accuracy: 97.62% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01, total= 14.2s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01 0 Validation loss: 0.077256 Best loss: 0.077256 Accuracy: 97.93% 1 Validation loss: 0.081539 Best loss: 0.077256 Accuracy: 98.20% 2 Validation loss: 0.083210 Best loss: 0.077256 Accuracy: 98.28% 3 Validation loss: 0.054373 Best loss: 0.054373 Accuracy: 98.71% 4 Validation loss: 0.050975 Best loss: 0.050975 Accuracy: 98.87% 5 Validation loss: 0.059530 Best loss: 0.050975 Accuracy: 98.67% 6 Validation loss: 0.058483 Best loss: 0.050975 Accuracy: 98.51% 7 Validation loss: 0.047411 Best loss: 0.047411 Accuracy: 98.63% 8 Validation loss: 0.049192 Best loss: 0.047411 Accuracy: 99.02% 9 Validation loss: 0.094554 Best loss: 0.047411 Accuracy: 98.28% 10 Validation loss: 0.177135 Best loss: 0.047411 Accuracy: 96.60% 11 Validation loss: 0.135463 Best loss: 0.047411 Accuracy: 98.55% 12 Validation loss: 0.126369 Best loss: 0.047411 Accuracy: 98.40% 13 Validation loss: 0.084567 Best loss: 0.047411 Accuracy: 98.44% 14 Validation loss: 0.102908 Best loss: 0.047411 Accuracy: 98.55% 15 Validation loss: 0.081483 Best loss: 0.047411 Accuracy: 98.71% 16 Validation loss: 0.074116 Best loss: 0.047411 Accuracy: 98.48% 17 Validation loss: 0.073612 Best loss: 0.047411 Accuracy: 98.71% 18 Validation loss: 0.075634 Best loss: 0.047411 Accuracy: 98.67% 19 Validation loss: 0.147183 Best loss: 0.047411 Accuracy: 98.75% 20 Validation loss: 0.043061 Best loss: 0.043061 Accuracy: 98.87% 21 Validation loss: 0.105160 Best loss: 0.043061 Accuracy: 98.98% 22 Validation loss: 0.065325 Best loss: 0.043061 Accuracy: 98.79% 23 Validation loss: 0.095793 Best loss: 0.043061 Accuracy: 98.05% 24 Validation loss: 0.128435 Best loss: 0.043061 Accuracy: 98.36% 25 Validation loss: 0.130819 Best loss: 0.043061 Accuracy: 98.48% 26 Validation loss: 0.131940 Best loss: 0.043061 Accuracy: 97.77% 27 Validation loss: 0.113928 Best loss: 0.043061 Accuracy: 97.77% 28 Validation loss: 0.097041 Best loss: 0.043061 Accuracy: 98.44% 29 Validation loss: 0.088444 Best loss: 0.043061 Accuracy: 98.44% 30 Validation loss: 0.194170 Best loss: 0.043061 Accuracy: 98.59% 31 Validation loss: 0.093873 Best loss: 0.043061 Accuracy: 98.32% 32 Validation loss: 0.176665 Best loss: 0.043061 Accuracy: 98.71% 33 Validation loss: 0.115290 Best loss: 0.043061 Accuracy: 98.40% 34 Validation loss: 0.210565 Best loss: 0.043061 Accuracy: 98.79% 35 Validation loss: 0.212074 Best loss: 0.043061 Accuracy: 98.55% 36 Validation loss: 2.096503 Best loss: 0.043061 Accuracy: 96.64% 37 Validation loss: 0.253084 Best loss: 0.043061 Accuracy: 95.74% 38 Validation loss: 0.593742 Best loss: 0.043061 Accuracy: 95.23% 39 Validation loss: 0.238404 Best loss: 0.043061 Accuracy: 97.22% 40 Validation loss: 0.170104 Best loss: 0.043061 Accuracy: 97.54% 41 Validation loss: 0.155084 Best loss: 0.043061 Accuracy: 98.28% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01, total= 16.9s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01 0 Validation loss: 0.093884 Best loss: 0.093884 Accuracy: 97.85% 1 Validation loss: 0.076332 Best loss: 0.076332 Accuracy: 97.97% 2 Validation loss: 0.085220 Best loss: 0.076332 Accuracy: 98.12% 3 Validation loss: 0.082966 Best loss: 0.076332 Accuracy: 98.05% 4 Validation loss: 0.057408 Best loss: 0.057408 Accuracy: 98.67% 5 Validation loss: 0.083013 Best loss: 0.057408 Accuracy: 98.24% 6 Validation loss: 0.072691 Best loss: 0.057408 Accuracy: 98.55% 7 Validation loss: 0.087917 Best loss: 0.057408 Accuracy: 98.05% 8 Validation loss: 0.132835 Best loss: 0.057408 Accuracy: 97.26% 9 Validation loss: 0.084901 Best loss: 0.057408 Accuracy: 98.40% 10 Validation loss: 0.079927 Best loss: 0.057408 Accuracy: 98.40% 11 Validation loss: 0.082857 Best loss: 0.057408 Accuracy: 98.40% 12 Validation loss: 0.088257 Best loss: 0.057408 Accuracy: 98.28% 13 Validation loss: 0.094664 Best loss: 0.057408 Accuracy: 98.59% 14 Validation loss: 0.074500 Best loss: 0.057408 Accuracy: 98.51% 15 Validation loss: 0.165562 Best loss: 0.057408 Accuracy: 97.30% 16 Validation loss: 0.091976 Best loss: 0.057408 Accuracy: 98.01% 17 Validation loss: 0.123025 Best loss: 0.057408 Accuracy: 97.69% 18 Validation loss: 0.106052 Best loss: 0.057408 Accuracy: 98.40% 19 Validation loss: 0.067828 Best loss: 0.057408 Accuracy: 98.55% 20 Validation loss: 0.095299 Best loss: 0.057408 Accuracy: 97.97% 21 Validation loss: 0.081483 Best loss: 0.057408 Accuracy: 98.63% 22 Validation loss: 0.094967 Best loss: 0.057408 Accuracy: 98.71% 23 Validation loss: 0.118287 Best loss: 0.057408 Accuracy: 98.63% 24 Validation loss: 0.073797 Best loss: 0.057408 Accuracy: 98.98% 25 Validation loss: 0.095701 Best loss: 0.057408 Accuracy: 98.75% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=100, learning_rate=0.01, total= 10.9s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01 0 Validation loss: 0.132217 Best loss: 0.132217 Accuracy: 96.33% 1 Validation loss: 0.542291 Best loss: 0.132217 Accuracy: 92.03% 2 Validation loss: 0.598429 Best loss: 0.132217 Accuracy: 73.85% 3 Validation loss: 0.811314 Best loss: 0.132217 Accuracy: 58.56% 4 Validation loss: 1.402884 Best loss: 0.132217 Accuracy: 55.59% 5 Validation loss: 0.852306 Best loss: 0.132217 Accuracy: 56.84% 6 Validation loss: 1.454235 Best loss: 0.132217 Accuracy: 53.91% 7 Validation loss: 1.583002 Best loss: 0.132217 Accuracy: 36.55% 8 Validation loss: 1.342375 Best loss: 0.132217 Accuracy: 33.19% 9 Validation loss: 1.539772 Best loss: 0.132217 Accuracy: 26.15% 10 Validation loss: 1.461075 Best loss: 0.132217 Accuracy: 30.38% 11 Validation loss: 1.381738 Best loss: 0.132217 Accuracy: 34.09% 12 Validation loss: 1.394869 Best loss: 0.132217 Accuracy: 33.54% 13 Validation loss: 1.611468 Best loss: 0.132217 Accuracy: 22.01% 14 Validation loss: 1.609337 Best loss: 0.132217 Accuracy: 22.01% 15 Validation loss: 1.609168 Best loss: 0.132217 Accuracy: 22.01% 16 Validation loss: 1.608323 Best loss: 0.132217 Accuracy: 22.01% 17 Validation loss: 1.608857 Best loss: 0.132217 Accuracy: 19.27% 18 Validation loss: 1.612009 Best loss: 0.132217 Accuracy: 22.01% 19 Validation loss: 1.609180 Best loss: 0.132217 Accuracy: 19.08% 20 Validation loss: 1.608321 Best loss: 0.132217 Accuracy: 20.91% 21 Validation loss: 1.613891 Best loss: 0.132217 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01, total= 1.3min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01 0 Validation loss: 0.150240 Best loss: 0.150240 Accuracy: 95.27% 1 Validation loss: 0.237669 Best loss: 0.150240 Accuracy: 94.68% 2 Validation loss: 0.188807 Best loss: 0.150240 Accuracy: 95.43% 3 Validation loss: 0.138834 Best loss: 0.138834 Accuracy: 96.52% 4 Validation loss: 0.312969 Best loss: 0.138834 Accuracy: 97.22% 5 Validation loss: 0.304368 Best loss: 0.138834 Accuracy: 91.05% 6 Validation loss: 0.416880 Best loss: 0.138834 Accuracy: 86.63% 7 Validation loss: 0.366710 Best loss: 0.138834 Accuracy: 89.52% 8 Validation loss: 0.419125 Best loss: 0.138834 Accuracy: 88.00% 9 Validation loss: 0.335204 Best loss: 0.138834 Accuracy: 94.88% 10 Validation loss: 0.348460 Best loss: 0.138834 Accuracy: 88.94% 11 Validation loss: 0.565747 Best loss: 0.138834 Accuracy: 72.63% 12 Validation loss: 0.443494 Best loss: 0.138834 Accuracy: 79.32% 13 Validation loss: 0.459230 Best loss: 0.138834 Accuracy: 78.54% 14 Validation loss: 0.483367 Best loss: 0.138834 Accuracy: 77.95% 15 Validation loss: 0.472077 Best loss: 0.138834 Accuracy: 78.30% 16 Validation loss: 0.548676 Best loss: 0.138834 Accuracy: 79.36% 17 Validation loss: 0.397524 Best loss: 0.138834 Accuracy: 79.16% 18 Validation loss: 0.584018 Best loss: 0.138834 Accuracy: 78.46% 19 Validation loss: 0.419598 Best loss: 0.138834 Accuracy: 78.58% 20 Validation loss: 2.175684 Best loss: 0.138834 Accuracy: 58.60% 21 Validation loss: 1.061435 Best loss: 0.138834 Accuracy: 39.68% 22 Validation loss: 1.045573 Best loss: 0.138834 Accuracy: 37.57% 23 Validation loss: 1.043344 Best loss: 0.138834 Accuracy: 42.14% 24 Validation loss: 1.031826 Best loss: 0.138834 Accuracy: 39.68% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01, total= 1.5min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01 0 Validation loss: 0.137977 Best loss: 0.137977 Accuracy: 97.11% 1 Validation loss: 0.163267 Best loss: 0.137977 Accuracy: 95.74% 2 Validation loss: 0.103080 Best loss: 0.103080 Accuracy: 96.91% 3 Validation loss: 0.540809 Best loss: 0.103080 Accuracy: 73.92% 4 Validation loss: 0.335075 Best loss: 0.103080 Accuracy: 93.00% 5 Validation loss: 0.250921 Best loss: 0.103080 Accuracy: 95.62% 6 Validation loss: 0.584629 Best loss: 0.103080 Accuracy: 94.33% 7 Validation loss: 0.175005 Best loss: 0.103080 Accuracy: 95.27% 8 Validation loss: 1.862055 Best loss: 0.103080 Accuracy: 58.72% 9 Validation loss: 0.786089 Best loss: 0.103080 Accuracy: 87.26% 10 Validation loss: 0.414133 Best loss: 0.103080 Accuracy: 88.27% 11 Validation loss: 0.788945 Best loss: 0.103080 Accuracy: 92.61% 12 Validation loss: 1.269745 Best loss: 0.103080 Accuracy: 40.11% 13 Validation loss: 1.579266 Best loss: 0.103080 Accuracy: 20.41% 14 Validation loss: 1.256671 Best loss: 0.103080 Accuracy: 39.84% 15 Validation loss: 1.403804 Best loss: 0.103080 Accuracy: 33.39% 16 Validation loss: 1.348766 Best loss: 0.103080 Accuracy: 34.68% 17 Validation loss: 1.341079 Best loss: 0.103080 Accuracy: 34.87% 18 Validation loss: 1.337494 Best loss: 0.103080 Accuracy: 34.68% 19 Validation loss: 1.342697 Best loss: 0.103080 Accuracy: 34.87% 20 Validation loss: 1.335060 Best loss: 0.103080 Accuracy: 34.83% 21 Validation loss: 1.351670 Best loss: 0.103080 Accuracy: 34.68% 22 Validation loss: 1.334777 Best loss: 0.103080 Accuracy: 34.68% 23 Validation loss: 1.334789 Best loss: 0.103080 Accuracy: 34.87% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=120, batch_size=10, learning_rate=0.01, total= 1.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1 0 Validation loss: 2253.869629 Best loss: 2253.869629 Accuracy: 19.47% 1 Validation loss: 197.237823 Best loss: 197.237823 Accuracy: 66.03% 2 Validation loss: 9.523738 Best loss: 9.523738 Accuracy: 91.71% 3 Validation loss: 6.686861 Best loss: 6.686861 Accuracy: 90.03% 4 Validation loss: 3.891983 Best loss: 3.891983 Accuracy: 91.91% 5 Validation loss: 2.482953 Best loss: 2.482953 Accuracy: 94.25% 6 Validation loss: 6.725624 Best loss: 2.482953 Accuracy: 84.28% 7 Validation loss: 2.146237 Best loss: 2.146237 Accuracy: 94.96% 8 Validation loss: 2.339620 Best loss: 2.146237 Accuracy: 93.04% 9 Validation loss: 2.990865 Best loss: 2.146237 Accuracy: 96.01% 10 Validation loss: 1.859377 Best loss: 1.859377 Accuracy: 95.97% 11 Validation loss: 1.278888 Best loss: 1.278888 Accuracy: 96.40% 12 Validation loss: 1.491071 Best loss: 1.278888 Accuracy: 95.11% 13 Validation loss: 1.020002 Best loss: 1.020002 Accuracy: 95.93% 14 Validation loss: 0.928930 Best loss: 0.928930 Accuracy: 96.13% 15 Validation loss: 1.238857 Best loss: 0.928930 Accuracy: 94.10% 16 Validation loss: 0.792561 Best loss: 0.792561 Accuracy: 95.78% 17 Validation loss: 1.875370 Best loss: 0.792561 Accuracy: 90.07% 18 Validation loss: 1.278079 Best loss: 0.792561 Accuracy: 94.72% 19 Validation loss: 0.577101 Best loss: 0.577101 Accuracy: 96.56% 20 Validation loss: 0.624225 Best loss: 0.577101 Accuracy: 96.13% 21 Validation loss: 0.830264 Best loss: 0.577101 Accuracy: 94.96% 22 Validation loss: 1.175743 Best loss: 0.577101 Accuracy: 93.12% 23 Validation loss: 0.581083 Best loss: 0.577101 Accuracy: 96.72% 24 Validation loss: 0.539774 Best loss: 0.539774 Accuracy: 96.72% 25 Validation loss: 0.724604 Best loss: 0.539774 Accuracy: 96.72% 26 Validation loss: 0.464204 Best loss: 0.464204 Accuracy: 97.07% 27 Validation loss: 0.442764 Best loss: 0.442764 Accuracy: 97.22% 28 Validation loss: 0.425368 Best loss: 0.425368 Accuracy: 96.79% 29 Validation loss: 0.420195 Best loss: 0.420195 Accuracy: 96.72% 30 Validation loss: 0.416196 Best loss: 0.416196 Accuracy: 96.95% 31 Validation loss: 0.332863 Best loss: 0.332863 Accuracy: 97.46% 32 Validation loss: 0.358017 Best loss: 0.332863 Accuracy: 97.54% 33 Validation loss: 0.378066 Best loss: 0.332863 Accuracy: 97.15% 34 Validation loss: 0.412616 Best loss: 0.332863 Accuracy: 96.40% 35 Validation loss: 0.339767 Best loss: 0.332863 Accuracy: 97.38% 36 Validation loss: 0.403039 Best loss: 0.332863 Accuracy: 97.11% 37 Validation loss: 0.370643 Best loss: 0.332863 Accuracy: 96.13% 38 Validation loss: 0.384398 Best loss: 0.332863 Accuracy: 97.38% 39 Validation loss: 0.402307 Best loss: 0.332863 Accuracy: 97.07% 40 Validation loss: 0.390818 Best loss: 0.332863 Accuracy: 97.07% 41 Validation loss: 0.387952 Best loss: 0.332863 Accuracy: 97.26% 42 Validation loss: 0.551784 Best loss: 0.332863 Accuracy: 96.72% 43 Validation loss: 0.329143 Best loss: 0.329143 Accuracy: 97.34% 44 Validation loss: 0.296295 Best loss: 0.296295 Accuracy: 97.46% 45 Validation loss: 0.844592 Best loss: 0.296295 Accuracy: 93.08% 46 Validation loss: 0.834674 Best loss: 0.296295 Accuracy: 96.36% 47 Validation loss: 0.672946 Best loss: 0.296295 Accuracy: 96.48% 48 Validation loss: 0.888101 Best loss: 0.296295 Accuracy: 95.50% 49 Validation loss: 0.420178 Best loss: 0.296295 Accuracy: 97.22% 50 Validation loss: 0.354646 Best loss: 0.296295 Accuracy: 97.62% 51 Validation loss: 0.544071 Best loss: 0.296295 Accuracy: 96.21% 52 Validation loss: 0.326440 Best loss: 0.296295 Accuracy: 97.50% 53 Validation loss: 0.321651 Best loss: 0.296295 Accuracy: 97.65% 54 Validation loss: 0.447301 Best loss: 0.296295 Accuracy: 97.26% 55 Validation loss: 0.319685 Best loss: 0.296295 Accuracy: 97.26% 56 Validation loss: 0.396502 Best loss: 0.296295 Accuracy: 97.07% 57 Validation loss: 0.518520 Best loss: 0.296295 Accuracy: 95.47% 58 Validation loss: 0.433026 Best loss: 0.296295 Accuracy: 97.42% 59 Validation loss: 0.353267 Best loss: 0.296295 Accuracy: 97.11% 60 Validation loss: 0.307492 Best loss: 0.296295 Accuracy: 97.50% 61 Validation loss: 0.637525 Best loss: 0.296295 Accuracy: 94.37% 62 Validation loss: 0.375093 Best loss: 0.296295 Accuracy: 97.34% 63 Validation loss: 0.409950 Best loss: 0.296295 Accuracy: 96.60% 64 Validation loss: 0.443079 Best loss: 0.296295 Accuracy: 96.40% 65 Validation loss: 0.594499 Best loss: 0.296295 Accuracy: 94.76% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1, total= 9.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1 0 Validation loss: 617.722412 Best loss: 617.722412 Accuracy: 48.91% 1 Validation loss: 44.148766 Best loss: 44.148766 Accuracy: 77.21% 2 Validation loss: 7.475353 Best loss: 7.475353 Accuracy: 85.85% 3 Validation loss: 2.200915 Best loss: 2.200915 Accuracy: 94.45% 4 Validation loss: 3.107678 Best loss: 2.200915 Accuracy: 88.15% 5 Validation loss: 3.270190 Best loss: 2.200915 Accuracy: 88.90% 6 Validation loss: 1.463994 Best loss: 1.463994 Accuracy: 95.93% 7 Validation loss: 2.309421 Best loss: 1.463994 Accuracy: 94.37% 8 Validation loss: 1.281925 Best loss: 1.281925 Accuracy: 96.05% 9 Validation loss: 1.151222 Best loss: 1.151222 Accuracy: 95.66% 10 Validation loss: 1.051753 Best loss: 1.051753 Accuracy: 96.48% 11 Validation loss: 0.922635 Best loss: 0.922635 Accuracy: 96.56% 12 Validation loss: 0.727884 Best loss: 0.727884 Accuracy: 96.48% 13 Validation loss: 0.836256 Best loss: 0.727884 Accuracy: 95.70% 14 Validation loss: 0.921362 Best loss: 0.727884 Accuracy: 95.58% 15 Validation loss: 0.787741 Best loss: 0.727884 Accuracy: 95.19% 16 Validation loss: 0.995767 Best loss: 0.727884 Accuracy: 94.25% 17 Validation loss: 1.086295 Best loss: 0.727884 Accuracy: 96.29% 18 Validation loss: 0.510853 Best loss: 0.510853 Accuracy: 97.30% 19 Validation loss: 0.701066 Best loss: 0.510853 Accuracy: 96.05% 20 Validation loss: 0.600601 Best loss: 0.510853 Accuracy: 96.13% 21 Validation loss: 0.672593 Best loss: 0.510853 Accuracy: 96.76% 22 Validation loss: 0.595011 Best loss: 0.510853 Accuracy: 97.15% 23 Validation loss: 0.867188 Best loss: 0.510853 Accuracy: 96.29% 24 Validation loss: 0.690836 Best loss: 0.510853 Accuracy: 96.99% 25 Validation loss: 0.627086 Best loss: 0.510853 Accuracy: 97.34% 26 Validation loss: 0.732949 Best loss: 0.510853 Accuracy: 95.31% 27 Validation loss: 0.553440 Best loss: 0.510853 Accuracy: 97.38% 28 Validation loss: 0.548715 Best loss: 0.510853 Accuracy: 97.46% 29 Validation loss: 0.532089 Best loss: 0.510853 Accuracy: 97.30% 30 Validation loss: 0.594906 Best loss: 0.510853 Accuracy: 97.42% 31 Validation loss: 0.644317 Best loss: 0.510853 Accuracy: 95.50% 32 Validation loss: 0.659883 Best loss: 0.510853 Accuracy: 96.25% 33 Validation loss: 1.202811 Best loss: 0.510853 Accuracy: 93.86% 34 Validation loss: 0.571845 Best loss: 0.510853 Accuracy: 97.26% 35 Validation loss: 0.438944 Best loss: 0.438944 Accuracy: 97.15% 36 Validation loss: 0.499801 Best loss: 0.438944 Accuracy: 96.95% 37 Validation loss: 0.406860 Best loss: 0.406860 Accuracy: 97.34% 38 Validation loss: 0.379837 Best loss: 0.379837 Accuracy: 97.38% 39 Validation loss: 0.504644 Best loss: 0.379837 Accuracy: 96.68% 40 Validation loss: 0.350152 Best loss: 0.350152 Accuracy: 97.69% 41 Validation loss: 0.572527 Best loss: 0.350152 Accuracy: 96.44% 42 Validation loss: 0.405733 Best loss: 0.350152 Accuracy: 97.58% 43 Validation loss: 0.483453 Best loss: 0.350152 Accuracy: 97.15% 44 Validation loss: 0.525112 Best loss: 0.350152 Accuracy: 97.54% 45 Validation loss: 0.528985 Best loss: 0.350152 Accuracy: 97.38% 46 Validation loss: 0.515326 Best loss: 0.350152 Accuracy: 96.91% 47 Validation loss: 0.627956 Best loss: 0.350152 Accuracy: 96.76% 48 Validation loss: 0.544897 Best loss: 0.350152 Accuracy: 97.19% 49 Validation loss: 0.557299 Best loss: 0.350152 Accuracy: 97.58% 50 Validation loss: 0.693732 Best loss: 0.350152 Accuracy: 96.87% 51 Validation loss: 1.155209 Best loss: 0.350152 Accuracy: 97.22% 52 Validation loss: 1.124076 Best loss: 0.350152 Accuracy: 97.73% 53 Validation loss: 0.922768 Best loss: 0.350152 Accuracy: 97.89% 54 Validation loss: 1.488722 Best loss: 0.350152 Accuracy: 97.73% 55 Validation loss: 1.471371 Best loss: 0.350152 Accuracy: 97.85% 56 Validation loss: 1.546350 Best loss: 0.350152 Accuracy: 97.54% 57 Validation loss: 2.008160 Best loss: 0.350152 Accuracy: 96.79% 58 Validation loss: 1.409400 Best loss: 0.350152 Accuracy: 97.22% 59 Validation loss: 1.654917 Best loss: 0.350152 Accuracy: 97.93% 60 Validation loss: 1.701032 Best loss: 0.350152 Accuracy: 96.68% 61 Validation loss: 1.100938 Best loss: 0.350152 Accuracy: 95.11% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1, total= 8.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1 0 Validation loss: 394.905090 Best loss: 394.905090 Accuracy: 58.09% 1 Validation loss: 15.409176 Best loss: 15.409176 Accuracy: 94.21% 2 Validation loss: 4.320905 Best loss: 4.320905 Accuracy: 95.39% 3 Validation loss: 2.062145 Best loss: 2.062145 Accuracy: 95.19% 4 Validation loss: 2.224840 Best loss: 2.062145 Accuracy: 96.13% 5 Validation loss: 1.533224 Best loss: 1.533224 Accuracy: 96.33% 6 Validation loss: 1.412018 Best loss: 1.412018 Accuracy: 96.64% 7 Validation loss: 1.413521 Best loss: 1.412018 Accuracy: 95.43% 8 Validation loss: 1.342562 Best loss: 1.342562 Accuracy: 95.70% 9 Validation loss: 1.180861 Best loss: 1.180861 Accuracy: 96.60% 10 Validation loss: 1.253484 Best loss: 1.180861 Accuracy: 96.52% 11 Validation loss: 1.313838 Best loss: 1.180861 Accuracy: 96.56% 12 Validation loss: 1.172045 Best loss: 1.172045 Accuracy: 96.17% 13 Validation loss: 1.263052 Best loss: 1.172045 Accuracy: 97.30% 14 Validation loss: 1.113346 Best loss: 1.113346 Accuracy: 95.97% 15 Validation loss: 1.039007 Best loss: 1.039007 Accuracy: 96.72% 16 Validation loss: 0.948227 Best loss: 0.948227 Accuracy: 96.87% 17 Validation loss: 1.068971 Best loss: 0.948227 Accuracy: 95.62% 18 Validation loss: 1.064954 Best loss: 0.948227 Accuracy: 97.19% 19 Validation loss: 1.163687 Best loss: 0.948227 Accuracy: 96.68% 20 Validation loss: 0.965772 Best loss: 0.948227 Accuracy: 97.03% 21 Validation loss: 1.000799 Best loss: 0.948227 Accuracy: 97.58% 22 Validation loss: 1.147547 Best loss: 0.948227 Accuracy: 96.87% 23 Validation loss: 0.962178 Best loss: 0.948227 Accuracy: 97.54% 24 Validation loss: 1.264516 Best loss: 0.948227 Accuracy: 96.99% 25 Validation loss: 0.935679 Best loss: 0.935679 Accuracy: 97.30% 26 Validation loss: 0.997939 Best loss: 0.935679 Accuracy: 97.03% 27 Validation loss: 0.977452 Best loss: 0.935679 Accuracy: 97.42% 28 Validation loss: 0.873381 Best loss: 0.873381 Accuracy: 97.42% 29 Validation loss: 0.547888 Best loss: 0.547888 Accuracy: 97.03% 30 Validation loss: 0.549266 Best loss: 0.547888 Accuracy: 97.34% 31 Validation loss: 0.521340 Best loss: 0.521340 Accuracy: 97.42% 32 Validation loss: 0.776709 Best loss: 0.521340 Accuracy: 96.09% 33 Validation loss: 0.666147 Best loss: 0.521340 Accuracy: 97.34% 34 Validation loss: 0.674552 Best loss: 0.521340 Accuracy: 97.89% 35 Validation loss: 0.656545 Best loss: 0.521340 Accuracy: 97.81% 36 Validation loss: 0.645453 Best loss: 0.521340 Accuracy: 97.46% 37 Validation loss: 0.743331 Best loss: 0.521340 Accuracy: 97.07% 38 Validation loss: 0.767466 Best loss: 0.521340 Accuracy: 96.68% 39 Validation loss: 0.755872 Best loss: 0.521340 Accuracy: 97.62% 40 Validation loss: 0.779493 Best loss: 0.521340 Accuracy: 96.79% 41 Validation loss: 0.724249 Best loss: 0.521340 Accuracy: 97.89% 42 Validation loss: 0.667347 Best loss: 0.521340 Accuracy: 97.34% 43 Validation loss: 0.804027 Best loss: 0.521340 Accuracy: 96.40% 44 Validation loss: 0.682661 Best loss: 0.521340 Accuracy: 97.85% 45 Validation loss: 0.737355 Best loss: 0.521340 Accuracy: 97.77% 46 Validation loss: 0.763166 Best loss: 0.521340 Accuracy: 97.97% 47 Validation loss: 0.799720 Best loss: 0.521340 Accuracy: 97.50% 48 Validation loss: 0.707011 Best loss: 0.521340 Accuracy: 98.08% 49 Validation loss: 0.773290 Best loss: 0.521340 Accuracy: 97.38% 50 Validation loss: 0.802538 Best loss: 0.521340 Accuracy: 97.85% 51 Validation loss: 0.726366 Best loss: 0.521340 Accuracy: 97.85% 52 Validation loss: 0.807451 Best loss: 0.521340 Accuracy: 98.01% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=100, batch_size=500, learning_rate=0.1, total= 7.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.104262 Best loss: 0.104262 Accuracy: 96.87% 1 Validation loss: 0.057768 Best loss: 0.057768 Accuracy: 98.20% 2 Validation loss: 0.065365 Best loss: 0.057768 Accuracy: 98.28% 3 Validation loss: 0.050140 Best loss: 0.050140 Accuracy: 98.55% 4 Validation loss: 0.059500 Best loss: 0.050140 Accuracy: 98.32% 5 Validation loss: 0.059232 Best loss: 0.050140 Accuracy: 98.16% 6 Validation loss: 0.050565 Best loss: 0.050140 Accuracy: 98.83% 7 Validation loss: 0.069235 Best loss: 0.050140 Accuracy: 98.28% 8 Validation loss: 0.057245 Best loss: 0.050140 Accuracy: 98.83% 9 Validation loss: 0.051185 Best loss: 0.050140 Accuracy: 98.83% 10 Validation loss: 0.063084 Best loss: 0.050140 Accuracy: 98.55% 11 Validation loss: 0.060704 Best loss: 0.050140 Accuracy: 98.71% 12 Validation loss: 0.071098 Best loss: 0.050140 Accuracy: 98.75% 13 Validation loss: 0.060864 Best loss: 0.050140 Accuracy: 98.63% 14 Validation loss: 0.112166 Best loss: 0.050140 Accuracy: 98.20% 15 Validation loss: 0.063503 Best loss: 0.050140 Accuracy: 98.36% 16 Validation loss: 0.061291 Best loss: 0.050140 Accuracy: 98.59% 17 Validation loss: 0.070368 Best loss: 0.050140 Accuracy: 98.75% 18 Validation loss: 0.089017 Best loss: 0.050140 Accuracy: 98.40% 19 Validation loss: 0.064526 Best loss: 0.050140 Accuracy: 98.83% 20 Validation loss: 0.054551 Best loss: 0.050140 Accuracy: 98.83% 21 Validation loss: 0.096824 Best loss: 0.050140 Accuracy: 98.63% 22 Validation loss: 0.066107 Best loss: 0.050140 Accuracy: 98.91% 23 Validation loss: 0.077998 Best loss: 0.050140 Accuracy: 98.40% 24 Validation loss: 0.099299 Best loss: 0.050140 Accuracy: 98.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 3.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.095852 Best loss: 0.095852 Accuracy: 96.72% 1 Validation loss: 0.066849 Best loss: 0.066849 Accuracy: 98.32% 2 Validation loss: 0.048709 Best loss: 0.048709 Accuracy: 98.59% 3 Validation loss: 0.051427 Best loss: 0.048709 Accuracy: 98.59% 4 Validation loss: 0.056287 Best loss: 0.048709 Accuracy: 98.40% 5 Validation loss: 0.050386 Best loss: 0.048709 Accuracy: 98.55% 6 Validation loss: 0.042117 Best loss: 0.042117 Accuracy: 98.87% 7 Validation loss: 0.051448 Best loss: 0.042117 Accuracy: 98.20% 8 Validation loss: 0.063650 Best loss: 0.042117 Accuracy: 98.48% 9 Validation loss: 0.056913 Best loss: 0.042117 Accuracy: 98.75% 10 Validation loss: 0.055860 Best loss: 0.042117 Accuracy: 98.98% 11 Validation loss: 0.055063 Best loss: 0.042117 Accuracy: 99.02% 12 Validation loss: 0.059420 Best loss: 0.042117 Accuracy: 98.67% 13 Validation loss: 0.037632 Best loss: 0.037632 Accuracy: 98.94% 14 Validation loss: 0.044770 Best loss: 0.037632 Accuracy: 99.10% 15 Validation loss: 0.067000 Best loss: 0.037632 Accuracy: 98.59% 16 Validation loss: 0.061861 Best loss: 0.037632 Accuracy: 98.79% 17 Validation loss: 0.060321 Best loss: 0.037632 Accuracy: 98.79% 18 Validation loss: 0.067400 Best loss: 0.037632 Accuracy: 98.59% 19 Validation loss: 0.076164 Best loss: 0.037632 Accuracy: 98.63% 20 Validation loss: 0.085849 Best loss: 0.037632 Accuracy: 98.59% 21 Validation loss: 0.090392 Best loss: 0.037632 Accuracy: 98.24% 22 Validation loss: 0.065538 Best loss: 0.037632 Accuracy: 98.32% 23 Validation loss: 0.082591 Best loss: 0.037632 Accuracy: 98.71% 24 Validation loss: 0.140387 Best loss: 0.037632 Accuracy: 98.16% 25 Validation loss: 0.120258 Best loss: 0.037632 Accuracy: 98.44% 26 Validation loss: 0.100676 Best loss: 0.037632 Accuracy: 98.59% 27 Validation loss: 0.098665 Best loss: 0.037632 Accuracy: 98.59% 28 Validation loss: 0.085320 Best loss: 0.037632 Accuracy: 98.24% 29 Validation loss: 0.127298 Best loss: 0.037632 Accuracy: 98.40% 30 Validation loss: 0.068495 Best loss: 0.037632 Accuracy: 98.87% 31 Validation loss: 0.085618 Best loss: 0.037632 Accuracy: 98.79% 32 Validation loss: 0.099967 Best loss: 0.037632 Accuracy: 98.67% 33 Validation loss: 69.966843 Best loss: 0.037632 Accuracy: 76.74% 34 Validation loss: 2.380670 Best loss: 0.037632 Accuracy: 94.25% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 5.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01 0 Validation loss: 0.092189 Best loss: 0.092189 Accuracy: 97.38% 1 Validation loss: 0.063842 Best loss: 0.063842 Accuracy: 98.36% 2 Validation loss: 0.055077 Best loss: 0.055077 Accuracy: 98.44% 3 Validation loss: 0.065851 Best loss: 0.055077 Accuracy: 98.40% 4 Validation loss: 0.047329 Best loss: 0.047329 Accuracy: 98.79% 5 Validation loss: 0.057213 Best loss: 0.047329 Accuracy: 98.44% 6 Validation loss: 0.052610 Best loss: 0.047329 Accuracy: 98.63% 7 Validation loss: 0.085337 Best loss: 0.047329 Accuracy: 98.05% 8 Validation loss: 0.063733 Best loss: 0.047329 Accuracy: 98.44% 9 Validation loss: 0.047242 Best loss: 0.047242 Accuracy: 98.67% 10 Validation loss: 0.053016 Best loss: 0.047242 Accuracy: 98.94% 11 Validation loss: 0.078120 Best loss: 0.047242 Accuracy: 98.44% 12 Validation loss: 0.066105 Best loss: 0.047242 Accuracy: 98.87% 13 Validation loss: 0.067185 Best loss: 0.047242 Accuracy: 98.67% 14 Validation loss: 0.075066 Best loss: 0.047242 Accuracy: 98.63% 15 Validation loss: 0.068906 Best loss: 0.047242 Accuracy: 98.67% 16 Validation loss: 0.066611 Best loss: 0.047242 Accuracy: 98.67% 17 Validation loss: 0.068865 Best loss: 0.047242 Accuracy: 98.83% 18 Validation loss: 0.054735 Best loss: 0.047242 Accuracy: 98.87% 19 Validation loss: 0.049881 Best loss: 0.047242 Accuracy: 98.94% 20 Validation loss: 0.056100 Best loss: 0.047242 Accuracy: 98.59% 21 Validation loss: 0.104917 Best loss: 0.047242 Accuracy: 98.67% 22 Validation loss: 0.098794 Best loss: 0.047242 Accuracy: 98.44% 23 Validation loss: 0.093185 Best loss: 0.047242 Accuracy: 98.75% 24 Validation loss: 0.075253 Best loss: 0.047242 Accuracy: 99.02% 25 Validation loss: 0.078304 Best loss: 0.047242 Accuracy: 98.63% 26 Validation loss: 0.139811 Best loss: 0.047242 Accuracy: 98.63% 27 Validation loss: 0.096919 Best loss: 0.047242 Accuracy: 98.63% 28 Validation loss: 0.096147 Best loss: 0.047242 Accuracy: 98.94% 29 Validation loss: 0.134537 Best loss: 0.047242 Accuracy: 98.79% 30 Validation loss: 0.081981 Best loss: 0.047242 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=140, batch_size=500, learning_rate=0.01, total= 4.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05 0 Validation loss: 1.805884 Best loss: 1.805884 Accuracy: 18.73% 1 Validation loss: 1.638453 Best loss: 1.638453 Accuracy: 20.91% 2 Validation loss: 1.777679 Best loss: 1.638453 Accuracy: 18.73% 3 Validation loss: 1.674783 Best loss: 1.638453 Accuracy: 19.27% 4 Validation loss: 1.644050 Best loss: 1.638453 Accuracy: 22.01% 5 Validation loss: 1.908150 Best loss: 1.638453 Accuracy: 19.08% 6 Validation loss: 1.861055 Best loss: 1.638453 Accuracy: 22.01% 7 Validation loss: 1.646454 Best loss: 1.638453 Accuracy: 19.27% 8 Validation loss: 1.682898 Best loss: 1.638453 Accuracy: 22.01% 9 Validation loss: 1.800022 Best loss: 1.638453 Accuracy: 18.73% 10 Validation loss: 1.787573 Best loss: 1.638453 Accuracy: 22.01% 11 Validation loss: 1.982571 Best loss: 1.638453 Accuracy: 22.01% 12 Validation loss: 1.943092 Best loss: 1.638453 Accuracy: 20.91% 13 Validation loss: 1.777602 Best loss: 1.638453 Accuracy: 19.27% 14 Validation loss: 1.791736 Best loss: 1.638453 Accuracy: 19.08% 15 Validation loss: 1.913676 Best loss: 1.638453 Accuracy: 22.01% 16 Validation loss: 1.850441 Best loss: 1.638453 Accuracy: 19.08% 17 Validation loss: 1.676427 Best loss: 1.638453 Accuracy: 20.91% 18 Validation loss: 1.803298 Best loss: 1.638453 Accuracy: 22.01% 19 Validation loss: 1.727334 Best loss: 1.638453 Accuracy: 22.01% 20 Validation loss: 1.709793 Best loss: 1.638453 Accuracy: 18.73% 21 Validation loss: 1.744467 Best loss: 1.638453 Accuracy: 18.73% 22 Validation loss: 1.651264 Best loss: 1.638453 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05, total= 1.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05 0 Validation loss: 1.746754 Best loss: 1.746754 Accuracy: 20.91% 1 Validation loss: 1.648125 Best loss: 1.648125 Accuracy: 19.08% 2 Validation loss: 1.792024 Best loss: 1.648125 Accuracy: 19.08% 3 Validation loss: 1.838272 Best loss: 1.648125 Accuracy: 19.27% 4 Validation loss: 1.800162 Best loss: 1.648125 Accuracy: 19.08% 5 Validation loss: 1.669709 Best loss: 1.648125 Accuracy: 18.73% 6 Validation loss: 1.851122 Best loss: 1.648125 Accuracy: 19.08% 7 Validation loss: 1.649280 Best loss: 1.648125 Accuracy: 19.27% 8 Validation loss: 1.899553 Best loss: 1.648125 Accuracy: 20.91% 9 Validation loss: 1.914811 Best loss: 1.648125 Accuracy: 19.27% 10 Validation loss: 1.686124 Best loss: 1.648125 Accuracy: 22.01% 11 Validation loss: 1.647474 Best loss: 1.647474 Accuracy: 20.91% 12 Validation loss: 1.853963 Best loss: 1.647474 Accuracy: 20.91% 13 Validation loss: 1.669570 Best loss: 1.647474 Accuracy: 22.01% 14 Validation loss: 1.703601 Best loss: 1.647474 Accuracy: 19.27% 15 Validation loss: 1.998326 Best loss: 1.647474 Accuracy: 19.08% 16 Validation loss: 1.656298 Best loss: 1.647474 Accuracy: 20.91% 17 Validation loss: 1.947681 Best loss: 1.647474 Accuracy: 20.91% 18 Validation loss: 2.143588 Best loss: 1.647474 Accuracy: 20.91% 19 Validation loss: 1.900223 Best loss: 1.647474 Accuracy: 20.91% 20 Validation loss: 1.930350 Best loss: 1.647474 Accuracy: 18.73% 21 Validation loss: 1.900985 Best loss: 1.647474 Accuracy: 22.01% 22 Validation loss: 1.750338 Best loss: 1.647474 Accuracy: 19.27% 23 Validation loss: 2.205183 Best loss: 1.647474 Accuracy: 18.73% 24 Validation loss: 1.718396 Best loss: 1.647474 Accuracy: 22.01% 25 Validation loss: 1.836215 Best loss: 1.647474 Accuracy: 22.01% 26 Validation loss: 1.777828 Best loss: 1.647474 Accuracy: 22.01% 27 Validation loss: 1.763804 Best loss: 1.647474 Accuracy: 19.27% 28 Validation loss: 1.860309 Best loss: 1.647474 Accuracy: 19.27% 29 Validation loss: 1.954945 Best loss: 1.647474 Accuracy: 18.73% 30 Validation loss: 1.838278 Best loss: 1.647474 Accuracy: 18.73% 31 Validation loss: 1.988403 Best loss: 1.647474 Accuracy: 22.01% 32 Validation loss: 2.001386 Best loss: 1.647474 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05, total= 1.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05 0 Validation loss: 1.648542 Best loss: 1.648542 Accuracy: 18.73% 1 Validation loss: 1.856751 Best loss: 1.648542 Accuracy: 19.08% 2 Validation loss: 1.736560 Best loss: 1.648542 Accuracy: 18.73% 3 Validation loss: 1.996893 Best loss: 1.648542 Accuracy: 19.08% 4 Validation loss: 2.140832 Best loss: 1.648542 Accuracy: 19.08% 5 Validation loss: 1.850734 Best loss: 1.648542 Accuracy: 19.27% 6 Validation loss: 1.746691 Best loss: 1.648542 Accuracy: 19.08% 7 Validation loss: 1.830478 Best loss: 1.648542 Accuracy: 18.73% 8 Validation loss: 1.649748 Best loss: 1.648542 Accuracy: 22.01% 9 Validation loss: 2.048924 Best loss: 1.648542 Accuracy: 22.01% 10 Validation loss: 1.832474 Best loss: 1.648542 Accuracy: 18.73% 11 Validation loss: 1.816107 Best loss: 1.648542 Accuracy: 19.27% 12 Validation loss: 1.836798 Best loss: 1.648542 Accuracy: 18.73% 13 Validation loss: 1.796182 Best loss: 1.648542 Accuracy: 19.08% 14 Validation loss: 1.792563 Best loss: 1.648542 Accuracy: 22.01% 15 Validation loss: 1.879796 Best loss: 1.648542 Accuracy: 20.91% 16 Validation loss: 1.704596 Best loss: 1.648542 Accuracy: 18.73% 17 Validation loss: 1.803578 Best loss: 1.648542 Accuracy: 19.08% 18 Validation loss: 2.113099 Best loss: 1.648542 Accuracy: 19.08% 19 Validation loss: 1.628995 Best loss: 1.628995 Accuracy: 19.27% 20 Validation loss: 1.801368 Best loss: 1.628995 Accuracy: 19.27% 21 Validation loss: 1.647708 Best loss: 1.628995 Accuracy: 22.01% 22 Validation loss: 1.653073 Best loss: 1.628995 Accuracy: 20.91% 23 Validation loss: 1.739612 Best loss: 1.628995 Accuracy: 22.01% 24 Validation loss: 1.861974 Best loss: 1.628995 Accuracy: 22.01% 25 Validation loss: 1.744545 Best loss: 1.628995 Accuracy: 19.27% 26 Validation loss: 1.719069 Best loss: 1.628995 Accuracy: 19.08% 27 Validation loss: 1.685109 Best loss: 1.628995 Accuracy: 22.01% 28 Validation loss: 2.323919 Best loss: 1.628995 Accuracy: 19.08% 29 Validation loss: 2.050603 Best loss: 1.628995 Accuracy: 18.73% 30 Validation loss: 1.854671 Best loss: 1.628995 Accuracy: 19.27% 31 Validation loss: 1.956271 Best loss: 1.628995 Accuracy: 18.73% 32 Validation loss: 1.666341 Best loss: 1.628995 Accuracy: 19.08% 33 Validation loss: 1.684591 Best loss: 1.628995 Accuracy: 20.91% 34 Validation loss: 1.747721 Best loss: 1.628995 Accuracy: 19.08% 35 Validation loss: 1.761389 Best loss: 1.628995 Accuracy: 22.01% 36 Validation loss: 1.971672 Best loss: 1.628995 Accuracy: 22.01% 37 Validation loss: 1.921872 Best loss: 1.628995 Accuracy: 22.01% 38 Validation loss: 1.989986 Best loss: 1.628995 Accuracy: 18.73% 39 Validation loss: 1.904658 Best loss: 1.628995 Accuracy: 19.08% 40 Validation loss: 1.730791 Best loss: 1.628995 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.05, total= 2.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05 0 Validation loss: 0.582922 Best loss: 0.582922 Accuracy: 87.26% 1 Validation loss: 700.583252 Best loss: 0.582922 Accuracy: 35.97% 2 Validation loss: 3.301509 Best loss: 0.582922 Accuracy: 60.28% 3 Validation loss: 1.445452 Best loss: 0.582922 Accuracy: 71.15% 4 Validation loss: 2.473347 Best loss: 0.582922 Accuracy: 55.94% 5 Validation loss: 1.350830 Best loss: 0.582922 Accuracy: 68.61% 6 Validation loss: 0.954446 Best loss: 0.582922 Accuracy: 78.42% 7 Validation loss: 0.790986 Best loss: 0.582922 Accuracy: 82.21% 8 Validation loss: 0.950953 Best loss: 0.582922 Accuracy: 71.31% 9 Validation loss: 0.702875 Best loss: 0.582922 Accuracy: 82.56% 10 Validation loss: 0.845586 Best loss: 0.582922 Accuracy: 75.92% 11 Validation loss: 0.696687 Best loss: 0.582922 Accuracy: 83.93% 12 Validation loss: 0.990190 Best loss: 0.582922 Accuracy: 85.81% 13 Validation loss: 1.144556 Best loss: 0.582922 Accuracy: 82.96% 14 Validation loss: 0.564097 Best loss: 0.564097 Accuracy: 87.26% 15 Validation loss: 0.645036 Best loss: 0.564097 Accuracy: 89.76% 16 Validation loss: 0.389776 Best loss: 0.389776 Accuracy: 91.67% 17 Validation loss: 0.392382 Best loss: 0.389776 Accuracy: 91.01% 18 Validation loss: 122089.085938 Best loss: 0.389776 Accuracy: 19.08% 19 Validation loss: 214.929688 Best loss: 0.389776 Accuracy: 58.01% 20 Validation loss: 308.215271 Best loss: 0.389776 Accuracy: 40.89% 21 Validation loss: 94.030769 Best loss: 0.389776 Accuracy: 67.24% 22 Validation loss: 420.204834 Best loss: 0.389776 Accuracy: 41.28% 23 Validation loss: 76.072571 Best loss: 0.389776 Accuracy: 67.40% 24 Validation loss: 131.180771 Best loss: 0.389776 Accuracy: 64.93% 25 Validation loss: 63.137894 Best loss: 0.389776 Accuracy: 75.18% 26 Validation loss: 82.910995 Best loss: 0.389776 Accuracy: 64.03% 27 Validation loss: 54.644066 Best loss: 0.389776 Accuracy: 79.67% 28 Validation loss: 58.119003 Best loss: 0.389776 Accuracy: 79.01% 29 Validation loss: 27.039621 Best loss: 0.389776 Accuracy: 79.98% 30 Validation loss: 40.613102 Best loss: 0.389776 Accuracy: 85.42% 31 Validation loss: 34.486137 Best loss: 0.389776 Accuracy: 84.56% 32 Validation loss: 92.991516 Best loss: 0.389776 Accuracy: 61.65% 33 Validation loss: 55.380478 Best loss: 0.389776 Accuracy: 69.78% 34 Validation loss: 51.647957 Best loss: 0.389776 Accuracy: 78.50% 35 Validation loss: 88.551918 Best loss: 0.389776 Accuracy: 78.93% 36 Validation loss: 28.808645 Best loss: 0.389776 Accuracy: 90.70% 37 Validation loss: 24012.804688 Best loss: 0.389776 Accuracy: 56.88% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05, total= 19.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05 0 Validation loss: 0.220876 Best loss: 0.220876 Accuracy: 93.47% 1 Validation loss: 74.324799 Best loss: 0.220876 Accuracy: 19.08% 2 Validation loss: 3.601294 Best loss: 0.220876 Accuracy: 56.22% 3 Validation loss: 2.934201 Best loss: 0.220876 Accuracy: 56.65% 4 Validation loss: 1.176287 Best loss: 0.220876 Accuracy: 77.60% 5 Validation loss: 1.054417 Best loss: 0.220876 Accuracy: 75.61% 6 Validation loss: 0.915681 Best loss: 0.220876 Accuracy: 80.49% 7 Validation loss: 0.539913 Best loss: 0.220876 Accuracy: 86.90% 8 Validation loss: 0.695490 Best loss: 0.220876 Accuracy: 85.65% 9 Validation loss: 0.865160 Best loss: 0.220876 Accuracy: 83.74% 10 Validation loss: 0.548597 Best loss: 0.220876 Accuracy: 86.36% 11 Validation loss: 0.562550 Best loss: 0.220876 Accuracy: 89.44% 12 Validation loss: 0.774707 Best loss: 0.220876 Accuracy: 85.46% 13 Validation loss: 0.727909 Best loss: 0.220876 Accuracy: 82.72% 14 Validation loss: 71.641373 Best loss: 0.220876 Accuracy: 34.99% 15 Validation loss: 3128.554443 Best loss: 0.220876 Accuracy: 14.50% 16 Validation loss: 2470.563965 Best loss: 0.220876 Accuracy: 18.49% 17 Validation loss: 2623.156982 Best loss: 0.220876 Accuracy: 19.04% 18 Validation loss: 564.759277 Best loss: 0.220876 Accuracy: 23.42% 19 Validation loss: 705.578613 Best loss: 0.220876 Accuracy: 20.09% 20 Validation loss: 118.579041 Best loss: 0.220876 Accuracy: 36.63% 21 Validation loss: 433.764648 Best loss: 0.220876 Accuracy: 24.78% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05, total= 11.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05 0 Validation loss: 0.904697 Best loss: 0.904697 Accuracy: 77.56% 1 Validation loss: 101.439499 Best loss: 0.904697 Accuracy: 19.27% 2 Validation loss: 6.030031 Best loss: 0.904697 Accuracy: 23.85% 3 Validation loss: 1.727677 Best loss: 0.904697 Accuracy: 44.64% 4 Validation loss: 2.114077 Best loss: 0.904697 Accuracy: 31.04% 5 Validation loss: 1.464838 Best loss: 0.904697 Accuracy: 57.08% 6 Validation loss: 4.117582 Best loss: 0.904697 Accuracy: 35.89% 7 Validation loss: 2.094471 Best loss: 0.904697 Accuracy: 48.87% 8 Validation loss: 0.971351 Best loss: 0.904697 Accuracy: 59.70% 9 Validation loss: 1.184114 Best loss: 0.904697 Accuracy: 57.97% 10 Validation loss: 0.832840 Best loss: 0.832840 Accuracy: 68.33% 11 Validation loss: 0.719886 Best loss: 0.719886 Accuracy: 69.78% 12 Validation loss: 0.739225 Best loss: 0.719886 Accuracy: 71.70% 13 Validation loss: 0.686744 Best loss: 0.686744 Accuracy: 70.76% 14 Validation loss: 1.025004 Best loss: 0.686744 Accuracy: 68.37% 15 Validation loss: 0.580116 Best loss: 0.580116 Accuracy: 78.23% 16 Validation loss: 0.485343 Best loss: 0.485343 Accuracy: 81.70% 17 Validation loss: 158464.562500 Best loss: 0.485343 Accuracy: 19.27% 18 Validation loss: 23701.232422 Best loss: 0.485343 Accuracy: 18.73% 19 Validation loss: 1056.956787 Best loss: 0.485343 Accuracy: 47.19% 20 Validation loss: 507.659241 Best loss: 0.485343 Accuracy: 56.22% 21 Validation loss: 709.591309 Best loss: 0.485343 Accuracy: 53.67% 22 Validation loss: 284.854675 Best loss: 0.485343 Accuracy: 58.56% 23 Validation loss: 400.561584 Best loss: 0.485343 Accuracy: 61.45% 24 Validation loss: 685.670715 Best loss: 0.485343 Accuracy: 66.26% 25 Validation loss: 163.521835 Best loss: 0.485343 Accuracy: 65.99% 26 Validation loss: 286.572845 Best loss: 0.485343 Accuracy: 64.74% 27 Validation loss: 288.016510 Best loss: 0.485343 Accuracy: 59.46% 28 Validation loss: 395.942780 Best loss: 0.485343 Accuracy: 66.73% 29 Validation loss: 424.118652 Best loss: 0.485343 Accuracy: 58.05% 30 Validation loss: 220.965744 Best loss: 0.485343 Accuracy: 67.40% 31 Validation loss: 292.131775 Best loss: 0.485343 Accuracy: 67.44% 32 Validation loss: 111.668259 Best loss: 0.485343 Accuracy: 65.09% 33 Validation loss: 209.823395 Best loss: 0.485343 Accuracy: 69.59% 34 Validation loss: 206.791199 Best loss: 0.485343 Accuracy: 59.19% 35 Validation loss: 117.022934 Best loss: 0.485343 Accuracy: 61.34% 36 Validation loss: 1190.013306 Best loss: 0.485343 Accuracy: 37.22% 37 Validation loss: 254.094635 Best loss: 0.485343 Accuracy: 53.48% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=100, learning_rate=0.05, total= 19.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02 0 Validation loss: 0.166459 Best loss: 0.166459 Accuracy: 96.72% 1 Validation loss: 0.349006 Best loss: 0.166459 Accuracy: 90.46% 2 Validation loss: 0.138778 Best loss: 0.138778 Accuracy: 96.13% 3 Validation loss: 0.130907 Best loss: 0.130907 Accuracy: 95.93% 4 Validation loss: 0.223436 Best loss: 0.130907 Accuracy: 95.58% 5 Validation loss: 6.558612 Best loss: 0.130907 Accuracy: 88.31% 6 Validation loss: 1.176257 Best loss: 0.130907 Accuracy: 92.77% 7 Validation loss: 0.863745 Best loss: 0.130907 Accuracy: 93.78% 8 Validation loss: 0.569394 Best loss: 0.130907 Accuracy: 95.86% 9 Validation loss: 0.420904 Best loss: 0.130907 Accuracy: 96.68% 10 Validation loss: 0.908756 Best loss: 0.130907 Accuracy: 95.74% 11 Validation loss: 0.437826 Best loss: 0.130907 Accuracy: 96.87% 12 Validation loss: 0.411749 Best loss: 0.130907 Accuracy: 97.30% 13 Validation loss: 0.513091 Best loss: 0.130907 Accuracy: 93.98% 14 Validation loss: 0.354452 Best loss: 0.130907 Accuracy: 96.64% 15 Validation loss: 0.608198 Best loss: 0.130907 Accuracy: 94.92% 16 Validation loss: 0.243279 Best loss: 0.130907 Accuracy: 97.46% 17 Validation loss: 14.964272 Best loss: 0.130907 Accuracy: 92.89% 18 Validation loss: 8.871965 Best loss: 0.130907 Accuracy: 94.33% 19 Validation loss: 3.344076 Best loss: 0.130907 Accuracy: 96.60% 20 Validation loss: 2.970726 Best loss: 0.130907 Accuracy: 95.90% 21 Validation loss: 2.974631 Best loss: 0.130907 Accuracy: 96.95% 22 Validation loss: 11.174094 Best loss: 0.130907 Accuracy: 92.42% 23 Validation loss: 2.135022 Best loss: 0.130907 Accuracy: 97.22% 24 Validation loss: 4.295427 Best loss: 0.130907 Accuracy: 95.47% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02, total= 24.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02 0 Validation loss: 0.650187 Best loss: 0.650187 Accuracy: 77.56% 1 Validation loss: 0.135336 Best loss: 0.135336 Accuracy: 97.30% 2 Validation loss: 0.137464 Best loss: 0.135336 Accuracy: 95.78% 3 Validation loss: 0.156157 Best loss: 0.135336 Accuracy: 95.90% 4 Validation loss: 0.118219 Best loss: 0.118219 Accuracy: 96.60% 5 Validation loss: 0.138770 Best loss: 0.118219 Accuracy: 97.15% 6 Validation loss: 0.101042 Best loss: 0.101042 Accuracy: 96.95% 7 Validation loss: 133.952606 Best loss: 0.101042 Accuracy: 81.59% 8 Validation loss: 2.094877 Best loss: 0.101042 Accuracy: 91.09% 9 Validation loss: 1.346749 Best loss: 0.101042 Accuracy: 95.74% 10 Validation loss: 0.895448 Best loss: 0.101042 Accuracy: 95.31% 11 Validation loss: 1.052235 Best loss: 0.101042 Accuracy: 94.14% 12 Validation loss: 1.325099 Best loss: 0.101042 Accuracy: 96.68% 13 Validation loss: 15.080662 Best loss: 0.101042 Accuracy: 95.23% 14 Validation loss: 2.057513 Best loss: 0.101042 Accuracy: 96.40% 15 Validation loss: 2.089115 Best loss: 0.101042 Accuracy: 95.15% 16 Validation loss: 5.047282 Best loss: 0.101042 Accuracy: 93.00% 17 Validation loss: 3.704069 Best loss: 0.101042 Accuracy: 95.19% 18 Validation loss: 2.047525 Best loss: 0.101042 Accuracy: 96.60% 19 Validation loss: 2.594218 Best loss: 0.101042 Accuracy: 95.54% 20 Validation loss: 1.936171 Best loss: 0.101042 Accuracy: 97.22% 21 Validation loss: 1.833137 Best loss: 0.101042 Accuracy: 96.95% 22 Validation loss: 1.065602 Best loss: 0.101042 Accuracy: 97.30% 23 Validation loss: 1.404787 Best loss: 0.101042 Accuracy: 96.52% 24 Validation loss: 1.703123 Best loss: 0.101042 Accuracy: 97.26% 25 Validation loss: 46.313206 Best loss: 0.101042 Accuracy: 92.57% 26 Validation loss: 4.116074 Best loss: 0.101042 Accuracy: 96.21% 27 Validation loss: 2.593573 Best loss: 0.101042 Accuracy: 97.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02, total= 26.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02 0 Validation loss: 0.265487 Best loss: 0.265487 Accuracy: 96.05% 1 Validation loss: 0.155474 Best loss: 0.155474 Accuracy: 97.22% 2 Validation loss: 0.088441 Best loss: 0.088441 Accuracy: 97.38% 3 Validation loss: 0.442180 Best loss: 0.088441 Accuracy: 89.60% 4 Validation loss: 0.339502 Best loss: 0.088441 Accuracy: 91.87% 5 Validation loss: 0.254256 Best loss: 0.088441 Accuracy: 92.81% 6 Validation loss: 0.204413 Best loss: 0.088441 Accuracy: 95.23% 7 Validation loss: 0.197526 Best loss: 0.088441 Accuracy: 95.11% 8 Validation loss: 0.132162 Best loss: 0.088441 Accuracy: 96.25% 9 Validation loss: 0.279311 Best loss: 0.088441 Accuracy: 93.67% 10 Validation loss: 0.151885 Best loss: 0.088441 Accuracy: 96.44% 11 Validation loss: 5.339515 Best loss: 0.088441 Accuracy: 79.52% 12 Validation loss: 1.184489 Best loss: 0.088441 Accuracy: 92.77% 13 Validation loss: 1.268098 Best loss: 0.088441 Accuracy: 95.35% 14 Validation loss: 0.848860 Best loss: 0.088441 Accuracy: 96.21% 15 Validation loss: 1.466763 Best loss: 0.088441 Accuracy: 92.57% 16 Validation loss: 1.163875 Best loss: 0.088441 Accuracy: 95.78% 17 Validation loss: 0.879525 Best loss: 0.088441 Accuracy: 96.60% 18 Validation loss: 0.365873 Best loss: 0.088441 Accuracy: 96.44% 19 Validation loss: 0.453455 Best loss: 0.088441 Accuracy: 94.68% 20 Validation loss: 0.556901 Best loss: 0.088441 Accuracy: 96.05% 21 Validation loss: 0.467962 Best loss: 0.088441 Accuracy: 93.16% 22 Validation loss: 0.288521 Best loss: 0.088441 Accuracy: 96.79% 23 Validation loss: 0.278377 Best loss: 0.088441 Accuracy: 96.68% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, n_neurons=100, batch_size=50, learning_rate=0.02, total= 22.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05 0 Validation loss: 2.086224 Best loss: 2.086224 Accuracy: 19.27% 1 Validation loss: 1.865216 Best loss: 1.865216 Accuracy: 22.01% 2 Validation loss: 1.800625 Best loss: 1.800625 Accuracy: 18.73% 3 Validation loss: 1.873947 Best loss: 1.800625 Accuracy: 19.08% 4 Validation loss: 1.731753 Best loss: 1.731753 Accuracy: 19.27% 5 Validation loss: 1.666483 Best loss: 1.666483 Accuracy: 19.08% 6 Validation loss: 2.220950 Best loss: 1.666483 Accuracy: 19.27% 7 Validation loss: 2.015726 Best loss: 1.666483 Accuracy: 19.08% 8 Validation loss: 2.090927 Best loss: 1.666483 Accuracy: 19.08% 9 Validation loss: 1.947276 Best loss: 1.666483 Accuracy: 22.01% 10 Validation loss: 1.792379 Best loss: 1.666483 Accuracy: 19.08% 11 Validation loss: 1.672828 Best loss: 1.666483 Accuracy: 19.08% 12 Validation loss: 1.912345 Best loss: 1.666483 Accuracy: 18.73% 13 Validation loss: 2.118674 Best loss: 1.666483 Accuracy: 19.08% 14 Validation loss: 2.516824 Best loss: 1.666483 Accuracy: 20.91% 15 Validation loss: 1.914143 Best loss: 1.666483 Accuracy: 22.01% 16 Validation loss: 2.225628 Best loss: 1.666483 Accuracy: 18.73% 17 Validation loss: 1.658233 Best loss: 1.658233 Accuracy: 20.91% 18 Validation loss: 2.221773 Best loss: 1.658233 Accuracy: 19.08% 19 Validation loss: 2.665043 Best loss: 1.658233 Accuracy: 22.01% 20 Validation loss: 2.008372 Best loss: 1.658233 Accuracy: 22.01% 21 Validation loss: 2.444497 Best loss: 1.658233 Accuracy: 20.91% 22 Validation loss: 2.119731 Best loss: 1.658233 Accuracy: 22.01% 23 Validation loss: 2.245319 Best loss: 1.658233 Accuracy: 19.27% 24 Validation loss: 1.920224 Best loss: 1.658233 Accuracy: 20.91% 25 Validation loss: 1.993324 Best loss: 1.658233 Accuracy: 20.91% 26 Validation loss: 1.718371 Best loss: 1.658233 Accuracy: 19.08% 27 Validation loss: 2.015471 Best loss: 1.658233 Accuracy: 18.73% 28 Validation loss: 2.296115 Best loss: 1.658233 Accuracy: 19.08% 29 Validation loss: 1.739970 Best loss: 1.658233 Accuracy: 20.91% 30 Validation loss: 1.709733 Best loss: 1.658233 Accuracy: 18.73% 31 Validation loss: 1.664723 Best loss: 1.658233 Accuracy: 20.91% 32 Validation loss: 1.779046 Best loss: 1.658233 Accuracy: 22.01% 33 Validation loss: 1.714498 Best loss: 1.658233 Accuracy: 19.08% 34 Validation loss: 2.160547 Best loss: 1.658233 Accuracy: 19.08% 35 Validation loss: 2.389714 Best loss: 1.658233 Accuracy: 19.27% 36 Validation loss: 1.843753 Best loss: 1.658233 Accuracy: 18.73% 37 Validation loss: 1.860697 Best loss: 1.658233 Accuracy: 19.08% 38 Validation loss: 2.205718 Best loss: 1.658233 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05, total= 2.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05 0 Validation loss: 1.889061 Best loss: 1.889061 Accuracy: 19.27% 1 Validation loss: 1.861101 Best loss: 1.861101 Accuracy: 22.01% 2 Validation loss: 2.010765 Best loss: 1.861101 Accuracy: 18.73% 3 Validation loss: 2.016443 Best loss: 1.861101 Accuracy: 22.01% 4 Validation loss: 1.970847 Best loss: 1.861101 Accuracy: 19.08% 5 Validation loss: 1.647392 Best loss: 1.647392 Accuracy: 20.91% 6 Validation loss: 1.701500 Best loss: 1.647392 Accuracy: 19.27% 7 Validation loss: 2.395791 Best loss: 1.647392 Accuracy: 19.08% 8 Validation loss: 1.941442 Best loss: 1.647392 Accuracy: 18.73% 9 Validation loss: 1.963343 Best loss: 1.647392 Accuracy: 20.91% 10 Validation loss: 1.835827 Best loss: 1.647392 Accuracy: 19.08% 11 Validation loss: 1.930027 Best loss: 1.647392 Accuracy: 22.01% 12 Validation loss: 1.885896 Best loss: 1.647392 Accuracy: 20.91% 13 Validation loss: 2.390907 Best loss: 1.647392 Accuracy: 19.08% 14 Validation loss: 1.961100 Best loss: 1.647392 Accuracy: 19.27% 15 Validation loss: 1.962754 Best loss: 1.647392 Accuracy: 20.91% 16 Validation loss: 1.886140 Best loss: 1.647392 Accuracy: 18.73% 17 Validation loss: 1.892863 Best loss: 1.647392 Accuracy: 19.08% 18 Validation loss: 1.981895 Best loss: 1.647392 Accuracy: 19.08% 19 Validation loss: 1.750127 Best loss: 1.647392 Accuracy: 22.01% 20 Validation loss: 2.134341 Best loss: 1.647392 Accuracy: 19.27% 21 Validation loss: 1.879304 Best loss: 1.647392 Accuracy: 22.01% 22 Validation loss: 2.148499 Best loss: 1.647392 Accuracy: 19.08% 23 Validation loss: 2.088906 Best loss: 1.647392 Accuracy: 22.01% 24 Validation loss: 1.838559 Best loss: 1.647392 Accuracy: 20.91% 25 Validation loss: 1.905608 Best loss: 1.647392 Accuracy: 19.08% 26 Validation loss: 2.406298 Best loss: 1.647392 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05, total= 1.6min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05 0 Validation loss: 1.831645 Best loss: 1.831645 Accuracy: 22.01% 1 Validation loss: 2.023754 Best loss: 1.831645 Accuracy: 19.27% 2 Validation loss: 1.934842 Best loss: 1.831645 Accuracy: 19.08% 3 Validation loss: 1.853903 Best loss: 1.831645 Accuracy: 19.27% 4 Validation loss: 2.686531 Best loss: 1.831645 Accuracy: 19.08% 5 Validation loss: 1.852321 Best loss: 1.831645 Accuracy: 20.91% 6 Validation loss: 1.812496 Best loss: 1.812496 Accuracy: 22.01% 7 Validation loss: 1.757771 Best loss: 1.757771 Accuracy: 19.08% 8 Validation loss: 1.902283 Best loss: 1.757771 Accuracy: 19.27% 9 Validation loss: 2.087057 Best loss: 1.757771 Accuracy: 19.27% 10 Validation loss: 1.779784 Best loss: 1.757771 Accuracy: 18.73% 11 Validation loss: 2.227396 Best loss: 1.757771 Accuracy: 19.27% 12 Validation loss: 1.950018 Best loss: 1.757771 Accuracy: 20.91% 13 Validation loss: 3.099452 Best loss: 1.757771 Accuracy: 19.08% 14 Validation loss: 1.830344 Best loss: 1.757771 Accuracy: 22.01% 15 Validation loss: 1.733783 Best loss: 1.733783 Accuracy: 20.91% 16 Validation loss: 2.189536 Best loss: 1.733783 Accuracy: 18.73% 17 Validation loss: 1.943920 Best loss: 1.733783 Accuracy: 18.73% 18 Validation loss: 1.848899 Best loss: 1.733783 Accuracy: 19.08% 19 Validation loss: 1.984283 Best loss: 1.733783 Accuracy: 18.73% 20 Validation loss: 1.919440 Best loss: 1.733783 Accuracy: 19.27% 21 Validation loss: 2.156083 Best loss: 1.733783 Accuracy: 22.01% 22 Validation loss: 1.834820 Best loss: 1.733783 Accuracy: 19.27% 23 Validation loss: 2.444077 Best loss: 1.733783 Accuracy: 19.08% 24 Validation loss: 1.866027 Best loss: 1.733783 Accuracy: 19.27% 25 Validation loss: 1.884747 Best loss: 1.733783 Accuracy: 22.01% 26 Validation loss: 1.738111 Best loss: 1.733783 Accuracy: 18.73% 27 Validation loss: 1.679295 Best loss: 1.679295 Accuracy: 20.91% 28 Validation loss: 2.567139 Best loss: 1.679295 Accuracy: 19.08% 29 Validation loss: 2.661161 Best loss: 1.679295 Accuracy: 20.91% 30 Validation loss: 2.013063 Best loss: 1.679295 Accuracy: 19.27% 31 Validation loss: 1.886540 Best loss: 1.679295 Accuracy: 22.01% 32 Validation loss: 1.891879 Best loss: 1.679295 Accuracy: 19.08% 33 Validation loss: 2.216809 Best loss: 1.679295 Accuracy: 22.01% 34 Validation loss: 2.191179 Best loss: 1.679295 Accuracy: 19.27% 35 Validation loss: 1.704169 Best loss: 1.679295 Accuracy: 19.08% 36 Validation loss: 1.731655 Best loss: 1.679295 Accuracy: 19.27% 37 Validation loss: 1.813348 Best loss: 1.679295 Accuracy: 22.01% 38 Validation loss: 3.032769 Best loss: 1.679295 Accuracy: 18.73% 39 Validation loss: 2.401283 Best loss: 1.679295 Accuracy: 19.27% 40 Validation loss: 1.645940 Best loss: 1.645940 Accuracy: 19.08% 41 Validation loss: 2.166197 Best loss: 1.645940 Accuracy: 22.01% 42 Validation loss: 2.384703 Best loss: 1.645940 Accuracy: 22.01% 43 Validation loss: 2.204781 Best loss: 1.645940 Accuracy: 20.91% 44 Validation loss: 1.765179 Best loss: 1.645940 Accuracy: 18.73% 45 Validation loss: 1.696824 Best loss: 1.645940 Accuracy: 18.73% 46 Validation loss: 2.521093 Best loss: 1.645940 Accuracy: 20.91% 47 Validation loss: 2.133249 Best loss: 1.645940 Accuracy: 19.08% 48 Validation loss: 1.695982 Best loss: 1.645940 Accuracy: 22.01% 49 Validation loss: 1.936504 Best loss: 1.645940 Accuracy: 19.08% 50 Validation loss: 2.176912 Best loss: 1.645940 Accuracy: 18.73% 51 Validation loss: 2.179085 Best loss: 1.645940 Accuracy: 19.27% 52 Validation loss: 2.013387 Best loss: 1.645940 Accuracy: 20.91% 53 Validation loss: 2.239875 Best loss: 1.645940 Accuracy: 19.27% 54 Validation loss: 1.942735 Best loss: 1.645940 Accuracy: 20.91% 55 Validation loss: 1.686159 Best loss: 1.645940 Accuracy: 20.91% 56 Validation loss: 2.281744 Best loss: 1.645940 Accuracy: 19.08% 57 Validation loss: 2.154368 Best loss: 1.645940 Accuracy: 22.01% 58 Validation loss: 1.815149 Best loss: 1.645940 Accuracy: 19.27% 59 Validation loss: 1.998513 Best loss: 1.645940 Accuracy: 19.27% 60 Validation loss: 1.969456 Best loss: 1.645940 Accuracy: 19.08% 61 Validation loss: 2.026058 Best loss: 1.645940 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=10, learning_rate=0.05, total= 3.7min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01 0 Validation loss: 0.089181 Best loss: 0.089181 Accuracy: 97.58% 1 Validation loss: 0.060156 Best loss: 0.060156 Accuracy: 97.97% 2 Validation loss: 0.071418 Best loss: 0.060156 Accuracy: 97.81% 3 Validation loss: 0.058439 Best loss: 0.058439 Accuracy: 98.01% 4 Validation loss: 0.067544 Best loss: 0.058439 Accuracy: 98.16% 5 Validation loss: 0.074274 Best loss: 0.058439 Accuracy: 97.93% 6 Validation loss: 0.052147 Best loss: 0.052147 Accuracy: 98.59% 7 Validation loss: 0.052012 Best loss: 0.052012 Accuracy: 98.63% 8 Validation loss: 0.051302 Best loss: 0.051302 Accuracy: 98.71% 9 Validation loss: 0.068325 Best loss: 0.051302 Accuracy: 98.44% 10 Validation loss: 0.065298 Best loss: 0.051302 Accuracy: 98.40% 11 Validation loss: 0.061077 Best loss: 0.051302 Accuracy: 98.55% 12 Validation loss: 0.059009 Best loss: 0.051302 Accuracy: 98.71% 13 Validation loss: 0.065330 Best loss: 0.051302 Accuracy: 98.55% 14 Validation loss: 0.095044 Best loss: 0.051302 Accuracy: 97.26% 15 Validation loss: 0.077465 Best loss: 0.051302 Accuracy: 98.20% 16 Validation loss: 0.060042 Best loss: 0.051302 Accuracy: 98.48% 17 Validation loss: 0.091115 Best loss: 0.051302 Accuracy: 98.44% 18 Validation loss: 0.079031 Best loss: 0.051302 Accuracy: 98.51% 19 Validation loss: 0.069121 Best loss: 0.051302 Accuracy: 98.67% 20 Validation loss: 0.070817 Best loss: 0.051302 Accuracy: 98.83% 21 Validation loss: 0.070062 Best loss: 0.051302 Accuracy: 98.94% 22 Validation loss: 0.077178 Best loss: 0.051302 Accuracy: 98.87% 23 Validation loss: 0.080524 Best loss: 0.051302 Accuracy: 98.87% 24 Validation loss: 0.082924 Best loss: 0.051302 Accuracy: 98.87% 25 Validation loss: 0.085960 Best loss: 0.051302 Accuracy: 98.87% 26 Validation loss: 0.087157 Best loss: 0.051302 Accuracy: 98.91% 27 Validation loss: 0.089623 Best loss: 0.051302 Accuracy: 98.87% 28 Validation loss: 0.090862 Best loss: 0.051302 Accuracy: 98.87% 29 Validation loss: 0.091876 Best loss: 0.051302 Accuracy: 98.87% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01, total= 4.2s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01 0 Validation loss: 0.096050 Best loss: 0.096050 Accuracy: 96.72% 1 Validation loss: 0.064769 Best loss: 0.064769 Accuracy: 98.36% 2 Validation loss: 0.058964 Best loss: 0.058964 Accuracy: 98.36% 3 Validation loss: 0.052890 Best loss: 0.052890 Accuracy: 98.67% 4 Validation loss: 0.067771 Best loss: 0.052890 Accuracy: 98.12% 5 Validation loss: 0.070698 Best loss: 0.052890 Accuracy: 97.81% 6 Validation loss: 0.054352 Best loss: 0.052890 Accuracy: 98.32% 7 Validation loss: 0.066684 Best loss: 0.052890 Accuracy: 98.28% 8 Validation loss: 0.062073 Best loss: 0.052890 Accuracy: 98.44% 9 Validation loss: 0.069366 Best loss: 0.052890 Accuracy: 98.44% 10 Validation loss: 0.075851 Best loss: 0.052890 Accuracy: 98.24% 11 Validation loss: 0.057570 Best loss: 0.052890 Accuracy: 98.20% 12 Validation loss: 0.068823 Best loss: 0.052890 Accuracy: 98.55% 13 Validation loss: 0.061140 Best loss: 0.052890 Accuracy: 98.91% 14 Validation loss: 0.083559 Best loss: 0.052890 Accuracy: 98.20% 15 Validation loss: 0.059797 Best loss: 0.052890 Accuracy: 98.67% 16 Validation loss: 0.056931 Best loss: 0.052890 Accuracy: 98.91% 17 Validation loss: 0.067920 Best loss: 0.052890 Accuracy: 98.48% 18 Validation loss: 0.065404 Best loss: 0.052890 Accuracy: 98.71% 19 Validation loss: 0.077042 Best loss: 0.052890 Accuracy: 98.71% 20 Validation loss: 0.063445 Best loss: 0.052890 Accuracy: 98.71% 21 Validation loss: 0.059130 Best loss: 0.052890 Accuracy: 98.79% 22 Validation loss: 0.071349 Best loss: 0.052890 Accuracy: 98.71% 23 Validation loss: 0.065831 Best loss: 0.052890 Accuracy: 98.94% 24 Validation loss: 0.069180 Best loss: 0.052890 Accuracy: 98.87% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01, total= 3.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01 0 Validation loss: 0.096636 Best loss: 0.096636 Accuracy: 97.19% 1 Validation loss: 0.069225 Best loss: 0.069225 Accuracy: 98.20% 2 Validation loss: 0.062340 Best loss: 0.062340 Accuracy: 98.16% 3 Validation loss: 0.058756 Best loss: 0.058756 Accuracy: 98.51% 4 Validation loss: 0.047036 Best loss: 0.047036 Accuracy: 98.63% 5 Validation loss: 0.047824 Best loss: 0.047036 Accuracy: 98.59% 6 Validation loss: 0.050137 Best loss: 0.047036 Accuracy: 98.71% 7 Validation loss: 0.069629 Best loss: 0.047036 Accuracy: 98.20% 8 Validation loss: 0.049101 Best loss: 0.047036 Accuracy: 98.51% 9 Validation loss: 0.050609 Best loss: 0.047036 Accuracy: 98.75% 10 Validation loss: 0.060328 Best loss: 0.047036 Accuracy: 98.36% 11 Validation loss: 0.054915 Best loss: 0.047036 Accuracy: 98.55% 12 Validation loss: 0.051907 Best loss: 0.047036 Accuracy: 98.59% 13 Validation loss: 0.061665 Best loss: 0.047036 Accuracy: 98.55% 14 Validation loss: 0.053320 Best loss: 0.047036 Accuracy: 98.59% 15 Validation loss: 0.039512 Best loss: 0.039512 Accuracy: 99.02% 16 Validation loss: 0.076026 Best loss: 0.039512 Accuracy: 98.55% 17 Validation loss: 0.051254 Best loss: 0.039512 Accuracy: 98.94% 18 Validation loss: 0.044463 Best loss: 0.039512 Accuracy: 99.06% 19 Validation loss: 0.052321 Best loss: 0.039512 Accuracy: 98.94% 20 Validation loss: 0.059585 Best loss: 0.039512 Accuracy: 98.71% 21 Validation loss: 0.058153 Best loss: 0.039512 Accuracy: 98.75% 22 Validation loss: 0.061467 Best loss: 0.039512 Accuracy: 98.87% 23 Validation loss: 0.043991 Best loss: 0.039512 Accuracy: 98.83% 24 Validation loss: 0.056272 Best loss: 0.039512 Accuracy: 98.83% 25 Validation loss: 0.060454 Best loss: 0.039512 Accuracy: 98.63% 26 Validation loss: 0.067750 Best loss: 0.039512 Accuracy: 98.55% 27 Validation loss: 0.055525 Best loss: 0.039512 Accuracy: 98.94% 28 Validation loss: 0.058377 Best loss: 0.039512 Accuracy: 99.02% 29 Validation loss: 0.063204 Best loss: 0.039512 Accuracy: 98.75% 30 Validation loss: 0.056829 Best loss: 0.039512 Accuracy: 98.98% 31 Validation loss: 0.077719 Best loss: 0.039512 Accuracy: 98.55% 32 Validation loss: 0.067430 Best loss: 0.039512 Accuracy: 98.94% 33 Validation loss: 0.049335 Best loss: 0.039512 Accuracy: 99.02% 34 Validation loss: 0.060056 Best loss: 0.039512 Accuracy: 98.87% 35 Validation loss: 0.050958 Best loss: 0.039512 Accuracy: 99.14% 36 Validation loss: 0.062346 Best loss: 0.039512 Accuracy: 98.91% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=50, batch_size=500, learning_rate=0.01, total= 4.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 1.723058 Best loss: 1.723058 Accuracy: 19.27% 1 Validation loss: 1.682333 Best loss: 1.682333 Accuracy: 19.08% 2 Validation loss: 1.648225 Best loss: 1.648225 Accuracy: 19.08% 3 Validation loss: 1.697291 Best loss: 1.648225 Accuracy: 18.73% 4 Validation loss: 1.674839 Best loss: 1.648225 Accuracy: 22.01% 5 Validation loss: 1.617249 Best loss: 1.617249 Accuracy: 20.91% 6 Validation loss: 1.653712 Best loss: 1.617249 Accuracy: 19.08% 7 Validation loss: 1.714154 Best loss: 1.617249 Accuracy: 18.73% 8 Validation loss: 1.716099 Best loss: 1.617249 Accuracy: 19.27% 9 Validation loss: 1.916621 Best loss: 1.617249 Accuracy: 20.91% 10 Validation loss: 1.749862 Best loss: 1.617249 Accuracy: 19.08% 11 Validation loss: 1.836787 Best loss: 1.617249 Accuracy: 18.73% 12 Validation loss: 1.745284 Best loss: 1.617249 Accuracy: 19.08% 13 Validation loss: 1.779952 Best loss: 1.617249 Accuracy: 20.91% 14 Validation loss: 1.660917 Best loss: 1.617249 Accuracy: 22.01% 15 Validation loss: 1.916949 Best loss: 1.617249 Accuracy: 18.73% 16 Validation loss: 1.648834 Best loss: 1.617249 Accuracy: 20.91% 17 Validation loss: 2.170762 Best loss: 1.617249 Accuracy: 20.91% 18 Validation loss: 1.753786 Best loss: 1.617249 Accuracy: 19.08% 19 Validation loss: 1.617513 Best loss: 1.617249 Accuracy: 19.08% 20 Validation loss: 1.740900 Best loss: 1.617249 Accuracy: 19.08% 21 Validation loss: 2.440020 Best loss: 1.617249 Accuracy: 20.91% 22 Validation loss: 1.700542 Best loss: 1.617249 Accuracy: 20.91% 23 Validation loss: 1.719535 Best loss: 1.617249 Accuracy: 20.91% 24 Validation loss: 1.821695 Best loss: 1.617249 Accuracy: 22.01% 25 Validation loss: 1.771733 Best loss: 1.617249 Accuracy: 19.27% 26 Validation loss: 1.814734 Best loss: 1.617249 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 20.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 1.661254 Best loss: 1.661254 Accuracy: 22.01% 1 Validation loss: 1.724407 Best loss: 1.661254 Accuracy: 19.08% 2 Validation loss: 1.619154 Best loss: 1.619154 Accuracy: 18.73% 3 Validation loss: 1.649606 Best loss: 1.619154 Accuracy: 22.01% 4 Validation loss: 1.636051 Best loss: 1.619154 Accuracy: 22.01% 5 Validation loss: 1.706603 Best loss: 1.619154 Accuracy: 20.91% 6 Validation loss: 1.688502 Best loss: 1.619154 Accuracy: 19.27% 7 Validation loss: 1.719708 Best loss: 1.619154 Accuracy: 18.73% 8 Validation loss: 1.658034 Best loss: 1.619154 Accuracy: 19.08% 9 Validation loss: 1.745630 Best loss: 1.619154 Accuracy: 22.01% 10 Validation loss: 1.728434 Best loss: 1.619154 Accuracy: 22.01% 11 Validation loss: 1.718781 Best loss: 1.619154 Accuracy: 22.01% 12 Validation loss: 1.659386 Best loss: 1.619154 Accuracy: 19.08% 13 Validation loss: 1.899715 Best loss: 1.619154 Accuracy: 18.73% 14 Validation loss: 1.749051 Best loss: 1.619154 Accuracy: 18.73% 15 Validation loss: 1.769904 Best loss: 1.619154 Accuracy: 18.73% 16 Validation loss: 1.656227 Best loss: 1.619154 Accuracy: 18.73% 17 Validation loss: 2.652435 Best loss: 1.619154 Accuracy: 19.27% 18 Validation loss: 1.866678 Best loss: 1.619154 Accuracy: 22.01% 19 Validation loss: 1.776570 Best loss: 1.619154 Accuracy: 19.27% 20 Validation loss: 1.831819 Best loss: 1.619154 Accuracy: 20.91% 21 Validation loss: 1.882256 Best loss: 1.619154 Accuracy: 22.01% 22 Validation loss: 2.168530 Best loss: 1.619154 Accuracy: 19.27% 23 Validation loss: 1.790612 Best loss: 1.619154 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 18.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1 0 Validation loss: 1.674267 Best loss: 1.674267 Accuracy: 22.01% 1 Validation loss: 1.637689 Best loss: 1.637689 Accuracy: 19.08% 2 Validation loss: 1.618065 Best loss: 1.618065 Accuracy: 22.01% 3 Validation loss: 1.668941 Best loss: 1.618065 Accuracy: 22.01% 4 Validation loss: 1.791728 Best loss: 1.618065 Accuracy: 19.27% 5 Validation loss: 1.776887 Best loss: 1.618065 Accuracy: 20.91% 6 Validation loss: 1.658970 Best loss: 1.618065 Accuracy: 22.01% 7 Validation loss: 1.683718 Best loss: 1.618065 Accuracy: 18.73% 8 Validation loss: 1.657858 Best loss: 1.618065 Accuracy: 18.73% 9 Validation loss: 1.795717 Best loss: 1.618065 Accuracy: 22.01% 10 Validation loss: 1.641840 Best loss: 1.618065 Accuracy: 18.73% 11 Validation loss: 1.786640 Best loss: 1.618065 Accuracy: 20.91% 12 Validation loss: 1.842332 Best loss: 1.618065 Accuracy: 19.08% 13 Validation loss: 1.751248 Best loss: 1.618065 Accuracy: 19.08% 14 Validation loss: 1.850258 Best loss: 1.618065 Accuracy: 22.01% 15 Validation loss: 1.868707 Best loss: 1.618065 Accuracy: 18.73% 16 Validation loss: 2.038511 Best loss: 1.618065 Accuracy: 19.08% 17 Validation loss: 2.272674 Best loss: 1.618065 Accuracy: 20.91% 18 Validation loss: 1.696373 Best loss: 1.618065 Accuracy: 22.01% 19 Validation loss: 1.695192 Best loss: 1.618065 Accuracy: 20.91% 20 Validation loss: 1.996304 Best loss: 1.618065 Accuracy: 20.91% 21 Validation loss: 1.856361 Best loss: 1.618065 Accuracy: 22.01% 22 Validation loss: 1.832013 Best loss: 1.618065 Accuracy: 19.27% 23 Validation loss: 1.718376 Best loss: 1.618065 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=50, learning_rate=0.1, total= 18.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02 0 Validation loss: 0.129327 Best loss: 0.129327 Accuracy: 95.93% 1 Validation loss: 0.111378 Best loss: 0.111378 Accuracy: 97.26% 2 Validation loss: 0.333454 Best loss: 0.111378 Accuracy: 95.35% 3 Validation loss: 1.719152 Best loss: 0.111378 Accuracy: 18.73% 4 Validation loss: 1.634094 Best loss: 0.111378 Accuracy: 22.01% 5 Validation loss: 1.697841 Best loss: 0.111378 Accuracy: 22.01% 6 Validation loss: 1.630629 Best loss: 0.111378 Accuracy: 19.27% 7 Validation loss: 1.659491 Best loss: 0.111378 Accuracy: 19.08% 8 Validation loss: 1.739564 Best loss: 0.111378 Accuracy: 19.27% 9 Validation loss: 1.870841 Best loss: 0.111378 Accuracy: 19.27% 10 Validation loss: 1.721948 Best loss: 0.111378 Accuracy: 22.01% 11 Validation loss: 1.762167 Best loss: 0.111378 Accuracy: 22.01% 12 Validation loss: 1.726558 Best loss: 0.111378 Accuracy: 19.08% 13 Validation loss: 1.682296 Best loss: 0.111378 Accuracy: 20.91% 14 Validation loss: 1.683757 Best loss: 0.111378 Accuracy: 22.01% 15 Validation loss: 1.652447 Best loss: 0.111378 Accuracy: 22.01% 16 Validation loss: 1.762875 Best loss: 0.111378 Accuracy: 18.73% 17 Validation loss: 1.647073 Best loss: 0.111378 Accuracy: 19.27% 18 Validation loss: 1.696715 Best loss: 0.111378 Accuracy: 20.91% 19 Validation loss: 1.637325 Best loss: 0.111378 Accuracy: 22.01% 20 Validation loss: 1.662044 Best loss: 0.111378 Accuracy: 19.27% 21 Validation loss: 1.722755 Best loss: 0.111378 Accuracy: 19.27% 22 Validation loss: 1.673080 Best loss: 0.111378 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02, total= 17.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02 0 Validation loss: 1.710925 Best loss: 1.710925 Accuracy: 19.08% 1 Validation loss: 1.621659 Best loss: 1.621659 Accuracy: 19.08% 2 Validation loss: 1.620910 Best loss: 1.620910 Accuracy: 19.27% 3 Validation loss: 1.684707 Best loss: 1.620910 Accuracy: 18.73% 4 Validation loss: 1.702600 Best loss: 1.620910 Accuracy: 19.27% 5 Validation loss: 1.664580 Best loss: 1.620910 Accuracy: 20.91% 6 Validation loss: 1.693564 Best loss: 1.620910 Accuracy: 19.27% 7 Validation loss: 1.841842 Best loss: 1.620910 Accuracy: 18.73% 8 Validation loss: 1.826492 Best loss: 1.620910 Accuracy: 18.73% 9 Validation loss: 1.751093 Best loss: 1.620910 Accuracy: 20.91% 10 Validation loss: 1.759145 Best loss: 1.620910 Accuracy: 19.08% 11 Validation loss: 1.666897 Best loss: 1.620910 Accuracy: 22.01% 12 Validation loss: 1.817140 Best loss: 1.620910 Accuracy: 20.91% 13 Validation loss: 1.667594 Best loss: 1.620910 Accuracy: 19.08% 14 Validation loss: 1.662121 Best loss: 1.620910 Accuracy: 19.08% 15 Validation loss: 1.737223 Best loss: 1.620910 Accuracy: 19.08% 16 Validation loss: 1.661655 Best loss: 1.620910 Accuracy: 18.73% 17 Validation loss: 1.656978 Best loss: 1.620910 Accuracy: 19.27% 18 Validation loss: 1.635023 Best loss: 1.620910 Accuracy: 20.91% 19 Validation loss: 1.649993 Best loss: 1.620910 Accuracy: 19.27% 20 Validation loss: 1.682832 Best loss: 1.620910 Accuracy: 18.73% 21 Validation loss: 1.770073 Best loss: 1.620910 Accuracy: 22.01% 22 Validation loss: 1.874124 Best loss: 1.620910 Accuracy: 18.73% 23 Validation loss: 1.726430 Best loss: 1.620910 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02, total= 18.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02 0 Validation loss: 0.112288 Best loss: 0.112288 Accuracy: 96.60% 1 Validation loss: 0.109388 Best loss: 0.109388 Accuracy: 97.38% 2 Validation loss: 1.746064 Best loss: 0.109388 Accuracy: 20.91% 3 Validation loss: 1.659012 Best loss: 0.109388 Accuracy: 22.01% 4 Validation loss: 1.745761 Best loss: 0.109388 Accuracy: 19.27% 5 Validation loss: 1.750450 Best loss: 0.109388 Accuracy: 20.91% 6 Validation loss: 1.658800 Best loss: 0.109388 Accuracy: 19.27% 7 Validation loss: 1.744601 Best loss: 0.109388 Accuracy: 18.73% 8 Validation loss: 1.715506 Best loss: 0.109388 Accuracy: 22.01% 9 Validation loss: 1.908716 Best loss: 0.109388 Accuracy: 20.91% 10 Validation loss: 1.648858 Best loss: 0.109388 Accuracy: 18.73% 11 Validation loss: 1.899889 Best loss: 0.109388 Accuracy: 18.73% 12 Validation loss: 1.806037 Best loss: 0.109388 Accuracy: 19.08% 13 Validation loss: 1.624468 Best loss: 0.109388 Accuracy: 22.01% 14 Validation loss: 1.684902 Best loss: 0.109388 Accuracy: 19.08% 15 Validation loss: 1.635304 Best loss: 0.109388 Accuracy: 22.01% 16 Validation loss: 1.627347 Best loss: 0.109388 Accuracy: 22.01% 17 Validation loss: 1.658033 Best loss: 0.109388 Accuracy: 18.73% 18 Validation loss: 1.639532 Best loss: 0.109388 Accuracy: 19.08% 19 Validation loss: 1.740164 Best loss: 0.109388 Accuracy: 18.73% 20 Validation loss: 1.720588 Best loss: 0.109388 Accuracy: 18.73% 21 Validation loss: 1.828202 Best loss: 0.109388 Accuracy: 22.01% 22 Validation loss: 1.670282 Best loss: 0.109388 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, learning_rate=0.02, total= 18.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1 0 Validation loss: 1.610068 Best loss: 1.610068 Accuracy: 22.01% 1 Validation loss: 1.618948 Best loss: 1.610068 Accuracy: 22.01% 2 Validation loss: 1.611184 Best loss: 1.610068 Accuracy: 22.01% 3 Validation loss: 1.608299 Best loss: 1.608299 Accuracy: 22.01% 4 Validation loss: 1.618208 Best loss: 1.608299 Accuracy: 22.01% 5 Validation loss: 1.615892 Best loss: 1.608299 Accuracy: 22.01% 6 Validation loss: 1.618005 Best loss: 1.608299 Accuracy: 22.01% 7 Validation loss: 1.617732 Best loss: 1.608299 Accuracy: 22.01% 8 Validation loss: 1.617135 Best loss: 1.608299 Accuracy: 19.27% 9 Validation loss: 1.608700 Best loss: 1.608299 Accuracy: 22.01% 10 Validation loss: 1.609198 Best loss: 1.608299 Accuracy: 22.01% 11 Validation loss: 1.608781 Best loss: 1.608299 Accuracy: 22.01% 12 Validation loss: 1.616520 Best loss: 1.608299 Accuracy: 22.01% 13 Validation loss: 1.625469 Best loss: 1.608299 Accuracy: 18.73% 14 Validation loss: 1.614392 Best loss: 1.608299 Accuracy: 18.73% 15 Validation loss: 1.612090 Best loss: 1.608299 Accuracy: 22.01% 16 Validation loss: 1.608089 Best loss: 1.608089 Accuracy: 22.01% 17 Validation loss: 1.615100 Best loss: 1.608089 Accuracy: 19.27% 18 Validation loss: 1.616772 Best loss: 1.608089 Accuracy: 18.73% 19 Validation loss: 1.612292 Best loss: 1.608089 Accuracy: 22.01% 20 Validation loss: 1.615546 Best loss: 1.608089 Accuracy: 19.08% 21 Validation loss: 1.619602 Best loss: 1.608089 Accuracy: 19.27% 22 Validation loss: 1.616395 Best loss: 1.608089 Accuracy: 18.73% 23 Validation loss: 1.615318 Best loss: 1.608089 Accuracy: 19.27% 24 Validation loss: 1.615881 Best loss: 1.608089 Accuracy: 22.01% 25 Validation loss: 1.613409 Best loss: 1.608089 Accuracy: 22.01% 26 Validation loss: 1.619709 Best loss: 1.608089 Accuracy: 19.08% 27 Validation loss: 1.620597 Best loss: 1.608089 Accuracy: 19.08% 28 Validation loss: 1.610771 Best loss: 1.608089 Accuracy: 19.08% 29 Validation loss: 1.610737 Best loss: 1.608089 Accuracy: 22.01% 30 Validation loss: 1.620082 Best loss: 1.608089 Accuracy: 19.27% 31 Validation loss: 1.614028 Best loss: 1.608089 Accuracy: 22.01% 32 Validation loss: 1.614077 Best loss: 1.608089 Accuracy: 22.01% 33 Validation loss: 1.615155 Best loss: 1.608089 Accuracy: 19.27% 34 Validation loss: 1.617152 Best loss: 1.608089 Accuracy: 22.01% 35 Validation loss: 1.615051 Best loss: 1.608089 Accuracy: 22.01% 36 Validation loss: 1.608877 Best loss: 1.608089 Accuracy: 20.91% 37 Validation loss: 1.611354 Best loss: 1.608089 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1, total= 29.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1 0 Validation loss: 0.623462 Best loss: 0.623462 Accuracy: 73.77% 1 Validation loss: 0.845186 Best loss: 0.623462 Accuracy: 58.91% 2 Validation loss: 0.282680 Best loss: 0.282680 Accuracy: 92.34% 3 Validation loss: 0.665196 Best loss: 0.282680 Accuracy: 78.77% 4 Validation loss: 0.275524 Best loss: 0.275524 Accuracy: 92.69% 5 Validation loss: 0.291226 Best loss: 0.275524 Accuracy: 92.42% 6 Validation loss: 0.281711 Best loss: 0.275524 Accuracy: 93.24% 7 Validation loss: 0.289921 Best loss: 0.275524 Accuracy: 93.67% 8 Validation loss: 1.613118 Best loss: 0.275524 Accuracy: 19.27% 9 Validation loss: 1.611726 Best loss: 0.275524 Accuracy: 22.01% 10 Validation loss: 1.609478 Best loss: 0.275524 Accuracy: 22.01% 11 Validation loss: 1.612491 Best loss: 0.275524 Accuracy: 22.01% 12 Validation loss: 1.616293 Best loss: 0.275524 Accuracy: 19.27% 13 Validation loss: 1.617469 Best loss: 0.275524 Accuracy: 19.08% 14 Validation loss: 1.610719 Best loss: 0.275524 Accuracy: 22.01% 15 Validation loss: 1.612067 Best loss: 0.275524 Accuracy: 22.01% 16 Validation loss: 1.619876 Best loss: 0.275524 Accuracy: 22.01% 17 Validation loss: 1.614757 Best loss: 0.275524 Accuracy: 19.08% 18 Validation loss: 1.611550 Best loss: 0.275524 Accuracy: 22.01% 19 Validation loss: 1.609965 Best loss: 0.275524 Accuracy: 22.01% 20 Validation loss: 1.608912 Best loss: 0.275524 Accuracy: 22.01% 21 Validation loss: 1.629909 Best loss: 0.275524 Accuracy: 22.01% 22 Validation loss: 1.620675 Best loss: 0.275524 Accuracy: 22.01% 23 Validation loss: 1.622632 Best loss: 0.275524 Accuracy: 22.01% 24 Validation loss: 1.607977 Best loss: 0.275524 Accuracy: 22.01% 25 Validation loss: 1.610930 Best loss: 0.275524 Accuracy: 20.91% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1, total= 20.6s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1 0 Validation loss: 1.611877 Best loss: 1.611877 Accuracy: 22.01% 1 Validation loss: 1.616043 Best loss: 1.611877 Accuracy: 19.27% 2 Validation loss: 1.613732 Best loss: 1.611877 Accuracy: 22.01% 3 Validation loss: 1.630102 Best loss: 1.611877 Accuracy: 18.73% 4 Validation loss: 1.608626 Best loss: 1.608626 Accuracy: 22.01% 5 Validation loss: 1.615780 Best loss: 1.608626 Accuracy: 22.01% 6 Validation loss: 1.615314 Best loss: 1.608626 Accuracy: 22.01% 7 Validation loss: 1.615385 Best loss: 1.608626 Accuracy: 18.73% 8 Validation loss: 1.612333 Best loss: 1.608626 Accuracy: 22.01% 9 Validation loss: 1.610839 Best loss: 1.608626 Accuracy: 22.01% 10 Validation loss: 1.611869 Best loss: 1.608626 Accuracy: 22.01% 11 Validation loss: 1.612863 Best loss: 1.608626 Accuracy: 22.01% 12 Validation loss: 1.610809 Best loss: 1.608626 Accuracy: 20.91% 13 Validation loss: 1.620051 Best loss: 1.608626 Accuracy: 19.27% 14 Validation loss: 1.612821 Best loss: 1.608626 Accuracy: 22.01% 15 Validation loss: 1.615504 Best loss: 1.608626 Accuracy: 22.01% 16 Validation loss: 1.628741 Best loss: 1.608626 Accuracy: 18.73% 17 Validation loss: 1.625350 Best loss: 1.608626 Accuracy: 19.08% 18 Validation loss: 1.609345 Best loss: 1.608626 Accuracy: 20.91% 19 Validation loss: 1.613122 Best loss: 1.608626 Accuracy: 19.08% 20 Validation loss: 1.613292 Best loss: 1.608626 Accuracy: 19.27% 21 Validation loss: 1.623134 Best loss: 1.608626 Accuracy: 22.01% 22 Validation loss: 1.617785 Best loss: 1.608626 Accuracy: 22.01% 23 Validation loss: 1.617092 Best loss: 1.608626 Accuracy: 22.01% 24 Validation loss: 1.608748 Best loss: 1.608626 Accuracy: 20.91% 25 Validation loss: 1.615600 Best loss: 1.608626 Accuracy: 18.73% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.1, total= 20.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1 0 Validation loss: 315.790131 Best loss: 315.790131 Accuracy: 66.42% 1 Validation loss: 11023.479492 Best loss: 315.790131 Accuracy: 52.66% 2 Validation loss: 4311.543945 Best loss: 315.790131 Accuracy: 73.30% 3 Validation loss: 13971.316406 Best loss: 315.790131 Accuracy: 60.13% 4 Validation loss: 1073.488892 Best loss: 315.790131 Accuracy: 77.25% 5 Validation loss: 753.452148 Best loss: 315.790131 Accuracy: 81.27% 6 Validation loss: 148.070724 Best loss: 148.070724 Accuracy: 89.87% 7 Validation loss: 133.770187 Best loss: 133.770187 Accuracy: 85.18% 8 Validation loss: 81.689438 Best loss: 81.689438 Accuracy: 92.26% 9 Validation loss: 81.466103 Best loss: 81.466103 Accuracy: 93.00% 10 Validation loss: 57.604755 Best loss: 57.604755 Accuracy: 93.94% 11 Validation loss: 48.464787 Best loss: 48.464787 Accuracy: 94.29% 12 Validation loss: 42.178226 Best loss: 42.178226 Accuracy: 93.75% 13 Validation loss: 49.435741 Best loss: 42.178226 Accuracy: 94.29% 14 Validation loss: 36.907036 Best loss: 36.907036 Accuracy: 94.61% 15 Validation loss: 32.451141 Best loss: 32.451141 Accuracy: 93.39% 16 Validation loss: 28.211035 Best loss: 28.211035 Accuracy: 93.94% 17 Validation loss: 41.058022 Best loss: 28.211035 Accuracy: 94.92% 18 Validation loss: 40.576786 Best loss: 28.211035 Accuracy: 93.12% 19 Validation loss: 39.429527 Best loss: 28.211035 Accuracy: 93.39% 20 Validation loss: 23.874954 Best loss: 23.874954 Accuracy: 95.78% 21 Validation loss: 21.163626 Best loss: 21.163626 Accuracy: 95.66% 22 Validation loss: 32.602306 Best loss: 21.163626 Accuracy: 93.12% 23 Validation loss: 23.363640 Best loss: 21.163626 Accuracy: 95.11% 24 Validation loss: 23.118732 Best loss: 21.163626 Accuracy: 94.25% 25 Validation loss: 24.013086 Best loss: 21.163626 Accuracy: 94.61% 26 Validation loss: 16.749643 Best loss: 16.749643 Accuracy: 95.50% 27 Validation loss: 19.638201 Best loss: 16.749643 Accuracy: 95.11% 28 Validation loss: 15.649062 Best loss: 15.649062 Accuracy: 95.54% 29 Validation loss: 24.214962 Best loss: 15.649062 Accuracy: 94.76% 30 Validation loss: 67.435181 Best loss: 15.649062 Accuracy: 94.45% 31 Validation loss: 40.282135 Best loss: 15.649062 Accuracy: 95.35% 32 Validation loss: 44.496750 Best loss: 15.649062 Accuracy: 93.78% 33 Validation loss: 35.451805 Best loss: 15.649062 Accuracy: 96.05% 34 Validation loss: 37.623798 Best loss: 15.649062 Accuracy: 95.04% 35 Validation loss: 29.930288 Best loss: 15.649062 Accuracy: 95.93% 36 Validation loss: 30.936785 Best loss: 15.649062 Accuracy: 96.01% 37 Validation loss: 26.974693 Best loss: 15.649062 Accuracy: 95.86% 38 Validation loss: 35.141434 Best loss: 15.649062 Accuracy: 94.96% 39 Validation loss: 22.056686 Best loss: 15.649062 Accuracy: 96.48% 40 Validation loss: 21.782885 Best loss: 15.649062 Accuracy: 96.01% 41 Validation loss: 26.714508 Best loss: 15.649062 Accuracy: 96.33% 42 Validation loss: 26.590208 Best loss: 15.649062 Accuracy: 95.39% 43 Validation loss: 19.656321 Best loss: 15.649062 Accuracy: 96.44% 44 Validation loss: 18.013765 Best loss: 15.649062 Accuracy: 96.33% 45 Validation loss: 15.815368 Best loss: 15.649062 Accuracy: 96.52% 46 Validation loss: 19.087854 Best loss: 15.649062 Accuracy: 95.43% 47 Validation loss: 22.944160 Best loss: 15.649062 Accuracy: 96.99% 48 Validation loss: 17.902771 Best loss: 15.649062 Accuracy: 95.93% 49 Validation loss: 19.295071 Best loss: 15.649062 Accuracy: 96.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1, total= 7.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1 0 Validation loss: 5893.396484 Best loss: 5893.396484 Accuracy: 22.01% 1 Validation loss: 29256.171875 Best loss: 5893.396484 Accuracy: 20.91% 2 Validation loss: 137876.890625 Best loss: 5893.396484 Accuracy: 19.27% 3 Validation loss: 4881.950195 Best loss: 4881.950195 Accuracy: 43.04% 4 Validation loss: 237.386429 Best loss: 237.386429 Accuracy: 74.75% 5 Validation loss: 129.066650 Best loss: 129.066650 Accuracy: 79.67% 6 Validation loss: 54.500965 Best loss: 54.500965 Accuracy: 88.74% 7 Validation loss: 57.459595 Best loss: 54.500965 Accuracy: 87.06% 8 Validation loss: 43.976082 Best loss: 43.976082 Accuracy: 89.25% 9 Validation loss: 47.188984 Best loss: 43.976082 Accuracy: 88.66% 10 Validation loss: 39.308830 Best loss: 39.308830 Accuracy: 90.07% 11 Validation loss: 31.216026 Best loss: 31.216026 Accuracy: 90.19% 12 Validation loss: 29.143494 Best loss: 29.143494 Accuracy: 89.41% 13 Validation loss: 95.607094 Best loss: 29.143494 Accuracy: 73.18% 14 Validation loss: 118.912262 Best loss: 29.143494 Accuracy: 82.06% 15 Validation loss: 58.726982 Best loss: 29.143494 Accuracy: 88.15% 16 Validation loss: 29.386679 Best loss: 29.143494 Accuracy: 91.87% 17 Validation loss: 47.779419 Best loss: 29.143494 Accuracy: 86.43% 18 Validation loss: 28.579145 Best loss: 28.579145 Accuracy: 91.95% 19 Validation loss: 37.749893 Best loss: 28.579145 Accuracy: 90.58% 20 Validation loss: 29.573847 Best loss: 28.579145 Accuracy: 91.79% 21 Validation loss: 31.001616 Best loss: 28.579145 Accuracy: 90.85% 22 Validation loss: 24.309494 Best loss: 24.309494 Accuracy: 90.50% 23 Validation loss: 18.902937 Best loss: 18.902937 Accuracy: 93.71% 24 Validation loss: 20.875706 Best loss: 18.902937 Accuracy: 91.99% 25 Validation loss: 16.206808 Best loss: 16.206808 Accuracy: 93.90% 26 Validation loss: 14.303451 Best loss: 14.303451 Accuracy: 93.39% 27 Validation loss: 1256.983398 Best loss: 14.303451 Accuracy: 49.26% 28 Validation loss: 60406032.000000 Best loss: 14.303451 Accuracy: 19.08% 29 Validation loss: 14858869.000000 Best loss: 14.303451 Accuracy: 18.73% 30 Validation loss: 777764.062500 Best loss: 14.303451 Accuracy: 63.49% 31 Validation loss: 64637.027344 Best loss: 14.303451 Accuracy: 74.28% 32 Validation loss: 69023.218750 Best loss: 14.303451 Accuracy: 73.85% 33 Validation loss: 25836.414062 Best loss: 14.303451 Accuracy: 81.39% 34 Validation loss: 20574.658203 Best loss: 14.303451 Accuracy: 81.27% 35 Validation loss: 20189.779297 Best loss: 14.303451 Accuracy: 82.13% 36 Validation loss: 19054.115234 Best loss: 14.303451 Accuracy: 83.11% 37 Validation loss: 23424.443359 Best loss: 14.303451 Accuracy: 78.81% 38 Validation loss: 24868.248047 Best loss: 14.303451 Accuracy: 80.14% 39 Validation loss: 16307.882812 Best loss: 14.303451 Accuracy: 84.28% 40 Validation loss: 14942.915039 Best loss: 14.303451 Accuracy: 80.45% 41 Validation loss: 11271.052734 Best loss: 14.303451 Accuracy: 84.64% 42 Validation loss: 12662.454102 Best loss: 14.303451 Accuracy: 82.92% 43 Validation loss: 35530.578125 Best loss: 14.303451 Accuracy: 77.37% 44 Validation loss: 43742.535156 Best loss: 14.303451 Accuracy: 73.61% 45 Validation loss: 11734.374023 Best loss: 14.303451 Accuracy: 85.07% 46 Validation loss: 9102.001953 Best loss: 14.303451 Accuracy: 86.63% 47 Validation loss: 9586.582031 Best loss: 14.303451 Accuracy: 85.97% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1, total= 7.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1 0 Validation loss: 67.551399 Best loss: 67.551399 Accuracy: 74.12% 1 Validation loss: 17.601042 Best loss: 17.601042 Accuracy: 77.44% 2 Validation loss: 13.156981 Best loss: 13.156981 Accuracy: 93.75% 3 Validation loss: 2.722600 Best loss: 2.722600 Accuracy: 92.69% 4 Validation loss: 6.160765 Best loss: 2.722600 Accuracy: 95.04% 5 Validation loss: 1.949417 Best loss: 1.949417 Accuracy: 95.43% 6 Validation loss: 1.461332 Best loss: 1.461332 Accuracy: 95.78% 7 Validation loss: 1.130372 Best loss: 1.130372 Accuracy: 93.75% 8 Validation loss: 0.827352 Best loss: 0.827352 Accuracy: 95.90% 9 Validation loss: 0.934506 Best loss: 0.827352 Accuracy: 95.74% 10 Validation loss: 0.766888 Best loss: 0.766888 Accuracy: 96.09% 11 Validation loss: 1.018077 Best loss: 0.766888 Accuracy: 95.97% 12 Validation loss: 0.909197 Best loss: 0.766888 Accuracy: 97.03% 13 Validation loss: 0.873174 Best loss: 0.766888 Accuracy: 95.93% 14 Validation loss: 1.783498 Best loss: 0.766888 Accuracy: 96.13% 15 Validation loss: 0.970142 Best loss: 0.766888 Accuracy: 96.76% 16 Validation loss: 0.658749 Best loss: 0.658749 Accuracy: 96.72% 17 Validation loss: 0.630908 Best loss: 0.630908 Accuracy: 96.79% 18 Validation loss: 0.678559 Best loss: 0.630908 Accuracy: 96.95% 19 Validation loss: 0.640590 Best loss: 0.630908 Accuracy: 96.95% 20 Validation loss: 0.784518 Best loss: 0.630908 Accuracy: 96.01% 21 Validation loss: 0.679451 Best loss: 0.630908 Accuracy: 96.91% 22 Validation loss: 0.609437 Best loss: 0.609437 Accuracy: 96.99% 23 Validation loss: 0.803975 Best loss: 0.609437 Accuracy: 95.39% 24 Validation loss: 0.700051 Best loss: 0.609437 Accuracy: 96.83% 25 Validation loss: 0.669092 Best loss: 0.609437 Accuracy: 97.07% 26 Validation loss: 0.592224 Best loss: 0.592224 Accuracy: 97.26% 27 Validation loss: 0.693349 Best loss: 0.592224 Accuracy: 96.64% 28 Validation loss: 0.723523 Best loss: 0.592224 Accuracy: 96.25% 29 Validation loss: 0.692990 Best loss: 0.592224 Accuracy: 96.01% 30 Validation loss: 0.736703 Best loss: 0.592224 Accuracy: 97.30% 31 Validation loss: 0.697325 Best loss: 0.592224 Accuracy: 96.60% 32 Validation loss: 0.921825 Best loss: 0.592224 Accuracy: 96.95% 33 Validation loss: 0.651796 Best loss: 0.592224 Accuracy: 95.19% 34 Validation loss: 0.557662 Best loss: 0.557662 Accuracy: 97.22% 35 Validation loss: 0.585079 Best loss: 0.557662 Accuracy: 97.03% 36 Validation loss: 0.589343 Best loss: 0.557662 Accuracy: 97.69% 37 Validation loss: 0.610641 Best loss: 0.557662 Accuracy: 97.30% 38 Validation loss: 0.647963 Best loss: 0.557662 Accuracy: 96.40% 39 Validation loss: 0.656247 Best loss: 0.557662 Accuracy: 96.76% 40 Validation loss: 0.851233 Best loss: 0.557662 Accuracy: 95.00% 41 Validation loss: 0.648111 Best loss: 0.557662 Accuracy: 96.48% 42 Validation loss: 0.660058 Best loss: 0.557662 Accuracy: 96.79% 43 Validation loss: 0.590463 Best loss: 0.557662 Accuracy: 97.26% 44 Validation loss: 0.773998 Best loss: 0.557662 Accuracy: 95.07% 45 Validation loss: 0.659962 Best loss: 0.557662 Accuracy: 97.62% 46 Validation loss: 0.860375 Best loss: 0.557662 Accuracy: 97.15% 47 Validation loss: 0.589291 Best loss: 0.557662 Accuracy: 97.58% 48 Validation loss: 0.665655 Best loss: 0.557662 Accuracy: 97.46% 49 Validation loss: 0.618193 Best loss: 0.557662 Accuracy: 96.72% 50 Validation loss: 0.559536 Best loss: 0.557662 Accuracy: 97.46% 51 Validation loss: 0.572969 Best loss: 0.557662 Accuracy: 98.01% 52 Validation loss: 0.700206 Best loss: 0.557662 Accuracy: 97.46% 53 Validation loss: 0.590519 Best loss: 0.557662 Accuracy: 97.42% 54 Validation loss: 0.673263 Best loss: 0.557662 Accuracy: 97.77% 55 Validation loss: 0.655309 Best loss: 0.557662 Accuracy: 96.99% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>, n_neurons=90, batch_size=500, learning_rate=0.1, total= 8.3s
[Parallel(n_jobs=1)]: Done 150 out of 150 | elapsed: 79.0min finished
0 Validation loss: 0.106986 Best loss: 0.106986 Accuracy: 96.52% 1 Validation loss: 0.075514 Best loss: 0.075514 Accuracy: 97.58% 2 Validation loss: 0.066724 Best loss: 0.066724 Accuracy: 97.85% 3 Validation loss: 0.077330 Best loss: 0.066724 Accuracy: 97.46% 4 Validation loss: 0.047736 Best loss: 0.047736 Accuracy: 98.40% 5 Validation loss: 0.089076 Best loss: 0.047736 Accuracy: 97.85% 6 Validation loss: 0.036265 Best loss: 0.036265 Accuracy: 98.94% 7 Validation loss: 0.037805 Best loss: 0.036265 Accuracy: 98.67% 8 Validation loss: 0.044360 Best loss: 0.036265 Accuracy: 98.63% 9 Validation loss: 0.040707 Best loss: 0.036265 Accuracy: 98.83% 10 Validation loss: 0.055964 Best loss: 0.036265 Accuracy: 98.67% 11 Validation loss: 0.053610 Best loss: 0.036265 Accuracy: 98.67% 12 Validation loss: 0.036851 Best loss: 0.036265 Accuracy: 99.14% 13 Validation loss: 0.059903 Best loss: 0.036265 Accuracy: 98.75% 14 Validation loss: 0.055655 Best loss: 0.036265 Accuracy: 98.87% 15 Validation loss: 0.052453 Best loss: 0.036265 Accuracy: 99.02% 16 Validation loss: 0.055165 Best loss: 0.036265 Accuracy: 98.94% 17 Validation loss: 0.052610 Best loss: 0.036265 Accuracy: 98.87% 18 Validation loss: 0.050384 Best loss: 0.036265 Accuracy: 98.94% 19 Validation loss: 0.040012 Best loss: 0.036265 Accuracy: 99.10% 20 Validation loss: 0.041638 Best loss: 0.036265 Accuracy: 99.02% 21 Validation loss: 0.037569 Best loss: 0.036265 Accuracy: 98.98% 22 Validation loss: 0.036688 Best loss: 0.036265 Accuracy: 99.10% 23 Validation loss: 0.080683 Best loss: 0.036265 Accuracy: 98.75% 24 Validation loss: 0.089796 Best loss: 0.036265 Accuracy: 98.32% 25 Validation loss: 0.067843 Best loss: 0.036265 Accuracy: 98.79% 26 Validation loss: 0.059075 Best loss: 0.036265 Accuracy: 98.28% 27 Validation loss: 0.044733 Best loss: 0.036265 Accuracy: 99.14% Early stopping!
Out[122]:
RandomizedSearchCV(cv=None, error_score='raise',
estimator=DNNClassifier(activation=<function elu at 0x7f95d738dbf8>,
batch_norm_momentum=None, batch_size=20, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=100,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42),
fit_params=None, iid=True, n_iter=50, n_jobs=1,
param_distributions={'activation': [<function relu at 0x7f95d7315f28>, <function elu at 0x7f95d738dbf8>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95c998e950>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f950b07b620>], 'n_neurons': [10, 30, 50, 70, 90, 100, 120, 140, 160], 'batch_size': [10, 50, 100, 500], 'learning_rate': [0.01, 0.02, 0.05, 0.1]},
pre_dispatch='2*n_jobs', random_state=42, refit=True,
return_train_score='warn', scoring=None, verbose=2)
In [123]:
rnd_search.best_params_
Out[123]:
{'activation': <function tensorflow.python.ops.gen_nn_ops.elu(features, name=None)>,
'batch_size': 500,
'learning_rate': 0.01,
'n_neurons': 140}
In [124]:
y_pred = rnd_search.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[124]:
0.9881299863786729
Wonderful! Tuning the hyperparameters got us up to 99.32% accuracy! It may not sound like a great improvement to go from 98.05% to 99.32% accuracy, but consider the error rate: it went from roughly 2% to 0.7%. That's a 65% reduction of the number of errors this model will produce!
It's a good idea to save this model:
In [125]:
rnd_search.best_estimator_.save("./my_best_mnist_model_0_to_4")
8.4.¶
Exercise: Now try adding Batch Normalization and compare the learning curves: is it converging faster than before? Does it produce a better model?
Let's train the best model found, once again, to see how fast it converges (alternatively, you could tweak the code above to make it write summaries for TensorBoard, so you can visualize the learning curve):
In [126]:
dnn_clf = DNNClassifier(activation=leaky_relu(alpha=0.1), batch_size=500, learning_rate=0.01,
n_neurons=140, random_state=42)
dnn_clf.fit(X_train1, y_train1, n_epochs=1000, X_valid=X_valid1, y_valid=y_valid1)
0 Validation loss: 0.068706 Best loss: 0.068706 Accuracy: 97.93% 1 Validation loss: 0.057475 Best loss: 0.057475 Accuracy: 98.48% 2 Validation loss: 0.044445 Best loss: 0.044445 Accuracy: 98.87% 3 Validation loss: 0.068179 Best loss: 0.044445 Accuracy: 98.36% 4 Validation loss: 0.040568 Best loss: 0.040568 Accuracy: 98.79% 5 Validation loss: 0.048001 Best loss: 0.040568 Accuracy: 98.71% 6 Validation loss: 0.043450 Best loss: 0.040568 Accuracy: 98.79% 7 Validation loss: 0.040337 Best loss: 0.040337 Accuracy: 98.83% 8 Validation loss: 0.050506 Best loss: 0.040337 Accuracy: 98.79% 9 Validation loss: 0.055531 Best loss: 0.040337 Accuracy: 98.71% 10 Validation loss: 0.049579 Best loss: 0.040337 Accuracy: 98.79% 11 Validation loss: 0.043808 Best loss: 0.040337 Accuracy: 98.98% 12 Validation loss: 0.046910 Best loss: 0.040337 Accuracy: 98.91% 13 Validation loss: 0.049127 Best loss: 0.040337 Accuracy: 98.87% 14 Validation loss: 0.051375 Best loss: 0.040337 Accuracy: 98.91% 15 Validation loss: 0.046793 Best loss: 0.040337 Accuracy: 98.91% 16 Validation loss: 0.053282 Best loss: 0.040337 Accuracy: 99.02% 17 Validation loss: 0.058460 Best loss: 0.040337 Accuracy: 99.22% 18 Validation loss: 0.051869 Best loss: 0.040337 Accuracy: 99.02% 19 Validation loss: 0.081018 Best loss: 0.040337 Accuracy: 98.51% 20 Validation loss: 0.057373 Best loss: 0.040337 Accuracy: 99.14% 21 Validation loss: 0.049985 Best loss: 0.040337 Accuracy: 99.14% 22 Validation loss: 0.059835 Best loss: 0.040337 Accuracy: 98.98% 23 Validation loss: 0.094914 Best loss: 0.040337 Accuracy: 98.98% 24 Validation loss: 2.428654 Best loss: 0.040337 Accuracy: 91.71% 25 Validation loss: 2.878971 Best loss: 0.040337 Accuracy: 95.47% 26 Validation loss: 0.499889 Best loss: 0.040337 Accuracy: 97.77% 27 Validation loss: 0.362998 Best loss: 0.040337 Accuracy: 97.62% 28 Validation loss: 0.302756 Best loss: 0.040337 Accuracy: 97.89% Early stopping!
Out[126]:
DNNClassifier(activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f9508fc7ea0>,
batch_norm_momentum=None, batch_size=500, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=140,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42)
The best loss is reached at epoch 19, but it was already within 10% of that result at epoch 9.
Let's check that we do indeed get 99.32% accuracy on the test set:
In [127]:
y_pred = dnn_clf.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[127]:
0.9877408056042032
Good, now let's use the exact same model, but this time with batch normalization:
In [128]:
dnn_clf_bn = DNNClassifier(activation=leaky_relu(alpha=0.1), batch_size=500, learning_rate=0.01,
n_neurons=90, random_state=42,
batch_norm_momentum=0.95)
dnn_clf_bn.fit(X_train1, y_train1, n_epochs=1000, X_valid=X_valid1, y_valid=y_valid1)
0 Validation loss: 0.050877 Best loss: 0.050877 Accuracy: 98.63% 1 Validation loss: 0.048369 Best loss: 0.048369 Accuracy: 98.48% 2 Validation loss: 0.045839 Best loss: 0.045839 Accuracy: 98.59% 3 Validation loss: 0.046061 Best loss: 0.045839 Accuracy: 98.75% 4 Validation loss: 0.062152 Best loss: 0.045839 Accuracy: 98.24% 5 Validation loss: 0.034786 Best loss: 0.034786 Accuracy: 98.94% 6 Validation loss: 0.034749 Best loss: 0.034749 Accuracy: 98.94% 7 Validation loss: 0.036506 Best loss: 0.034749 Accuracy: 98.98% 8 Validation loss: 0.049897 Best loss: 0.034749 Accuracy: 98.63% 9 Validation loss: 0.037980 Best loss: 0.034749 Accuracy: 98.94% 10 Validation loss: 0.040320 Best loss: 0.034749 Accuracy: 98.98% 11 Validation loss: 0.040401 Best loss: 0.034749 Accuracy: 98.87% 12 Validation loss: 0.039246 Best loss: 0.034749 Accuracy: 99.22% 13 Validation loss: 0.055425 Best loss: 0.034749 Accuracy: 98.55% 14 Validation loss: 0.042217 Best loss: 0.034749 Accuracy: 98.94% 15 Validation loss: 0.047152 Best loss: 0.034749 Accuracy: 99.02% 16 Validation loss: 0.038347 Best loss: 0.034749 Accuracy: 99.10% 17 Validation loss: 0.035724 Best loss: 0.034749 Accuracy: 99.10% 18 Validation loss: 0.035966 Best loss: 0.034749 Accuracy: 99.14% 19 Validation loss: 0.041954 Best loss: 0.034749 Accuracy: 99.02% 20 Validation loss: 0.039578 Best loss: 0.034749 Accuracy: 98.98% 21 Validation loss: 0.032711 Best loss: 0.032711 Accuracy: 99.18% 22 Validation loss: 0.040996 Best loss: 0.032711 Accuracy: 99.02% 23 Validation loss: 0.050894 Best loss: 0.032711 Accuracy: 99.02% 24 Validation loss: 0.055598 Best loss: 0.032711 Accuracy: 98.75% 25 Validation loss: 0.055539 Best loss: 0.032711 Accuracy: 98.98% 26 Validation loss: 0.039205 Best loss: 0.032711 Accuracy: 99.14% 27 Validation loss: 0.035096 Best loss: 0.032711 Accuracy: 98.94% 28 Validation loss: 0.042966 Best loss: 0.032711 Accuracy: 98.91% 29 Validation loss: 0.043835 Best loss: 0.032711 Accuracy: 99.06% 30 Validation loss: 0.035313 Best loss: 0.032711 Accuracy: 99.18% 31 Validation loss: 0.047484 Best loss: 0.032711 Accuracy: 99.06% 32 Validation loss: 0.047193 Best loss: 0.032711 Accuracy: 99.10% 33 Validation loss: 0.045797 Best loss: 0.032711 Accuracy: 99.10% 34 Validation loss: 0.040975 Best loss: 0.032711 Accuracy: 99.30% 35 Validation loss: 0.051753 Best loss: 0.032711 Accuracy: 99.02% 36 Validation loss: 0.052876 Best loss: 0.032711 Accuracy: 99.10% 37 Validation loss: 0.051288 Best loss: 0.032711 Accuracy: 99.06% 38 Validation loss: 0.049453 Best loss: 0.032711 Accuracy: 99.02% 39 Validation loss: 0.058928 Best loss: 0.032711 Accuracy: 98.98% 40 Validation loss: 0.065791 Best loss: 0.032711 Accuracy: 98.67% 41 Validation loss: 0.049721 Best loss: 0.032711 Accuracy: 98.75% 42 Validation loss: 0.043183 Best loss: 0.032711 Accuracy: 99.02% Early stopping!
Out[128]:
DNNClassifier(activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f95039a00d0>,
batch_norm_momentum=0.95, batch_size=500, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=90,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42)
The best params are reached during epoch 48, that's actually a slower convergence than earlier. Let's check the accuracy:
In [129]:
y_pred = dnn_clf_bn.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[129]:
0.9902704806382565
Well, batch normalization did not improve accuracy. Let's see if we can find a good set of hyperparameters that will work well with batch normalization:
In [130]:
from sklearn.model_selection import RandomizedSearchCV
param_distribs = {
"n_neurons": [10, 30, 50, 70, 90, 100, 120, 140, 160],
"batch_size": [10, 50, 100, 500],
"learning_rate": [0.01, 0.02, 0.05, 0.1],
"activation": [tf.nn.relu, tf.nn.elu, leaky_relu(alpha=0.01), leaky_relu(alpha=0.1)],
# you could also try exploring different numbers of hidden layers, different optimizers, etc.
#"n_hidden_layers": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
#"optimizer_class": [tf.train.AdamOptimizer, partial(tf.train.MomentumOptimizer, momentum=0.95)],
"batch_norm_momentum": [0.9, 0.95, 0.98, 0.99, 0.999],
}
rnd_search_bn = RandomizedSearchCV(DNNClassifier(random_state=42), param_distribs, n_iter=50,
fit_params={"X_valid": X_valid1, "y_valid": y_valid1, "n_epochs": 1000},
random_state=42, verbose=2)
rnd_search_bn.fit(X_train1, y_train1)
Fitting 3 folds for each of 50 candidates, totalling 150 fits [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01
/home/ageron/.virtualenvs/ml/lib/python3.5/site-packages/sklearn/model_selection/_search.py:584: DeprecationWarning: "fit_params" as a constructor argument was deprecated in version 0.19 and will be removed in version 0.21. Pass fit parameters to the "fit" method instead. '"fit" method instead.', DeprecationWarning)
0 Validation loss: 0.083536 Best loss: 0.083536 Accuracy: 97.69% 1 Validation loss: 0.074428 Best loss: 0.074428 Accuracy: 98.16% 2 Validation loss: 0.053495 Best loss: 0.053495 Accuracy: 98.51% 3 Validation loss: 0.050038 Best loss: 0.050038 Accuracy: 98.55% 4 Validation loss: 0.049013 Best loss: 0.049013 Accuracy: 98.98% 5 Validation loss: 0.039150 Best loss: 0.039150 Accuracy: 98.98% 6 Validation loss: 0.038751 Best loss: 0.038751 Accuracy: 98.87% 7 Validation loss: 0.085989 Best loss: 0.038751 Accuracy: 98.01% 8 Validation loss: 0.038020 Best loss: 0.038020 Accuracy: 98.94% 9 Validation loss: 0.050611 Best loss: 0.038020 Accuracy: 98.59% 10 Validation loss: 0.049003 Best loss: 0.038020 Accuracy: 98.67% 11 Validation loss: 0.043646 Best loss: 0.038020 Accuracy: 98.79% 12 Validation loss: 0.048067 Best loss: 0.038020 Accuracy: 98.63% 13 Validation loss: 0.052451 Best loss: 0.038020 Accuracy: 98.87% 14 Validation loss: 0.037606 Best loss: 0.037606 Accuracy: 98.87% 15 Validation loss: 0.048493 Best loss: 0.037606 Accuracy: 98.83% 16 Validation loss: 0.042378 Best loss: 0.037606 Accuracy: 99.10% 17 Validation loss: 0.048708 Best loss: 0.037606 Accuracy: 98.87% 18 Validation loss: 0.041217 Best loss: 0.037606 Accuracy: 98.91% 19 Validation loss: 0.040775 Best loss: 0.037606 Accuracy: 98.94% 20 Validation loss: 0.056285 Best loss: 0.037606 Accuracy: 98.71% 21 Validation loss: 0.052996 Best loss: 0.037606 Accuracy: 98.83% 22 Validation loss: 0.041326 Best loss: 0.037606 Accuracy: 99.14% 23 Validation loss: 0.048796 Best loss: 0.037606 Accuracy: 98.91% 24 Validation loss: 0.040158 Best loss: 0.037606 Accuracy: 99.02% 25 Validation loss: 0.037767 Best loss: 0.037606 Accuracy: 99.22% 26 Validation loss: 0.034669 Best loss: 0.034669 Accuracy: 99.02% 27 Validation loss: 0.041303 Best loss: 0.034669 Accuracy: 99.06% 28 Validation loss: 0.053501 Best loss: 0.034669 Accuracy: 98.87% 29 Validation loss: 0.053688 Best loss: 0.034669 Accuracy: 98.71% 30 Validation loss: 0.041105 Best loss: 0.034669 Accuracy: 99.10% 31 Validation loss: 0.054113 Best loss: 0.034669 Accuracy: 98.75% 32 Validation loss: 0.049274 Best loss: 0.034669 Accuracy: 98.91% 33 Validation loss: 0.043061 Best loss: 0.034669 Accuracy: 98.98% 34 Validation loss: 0.065145 Best loss: 0.034669 Accuracy: 98.63% 35 Validation loss: 0.038252 Best loss: 0.034669 Accuracy: 98.98% 36 Validation loss: 0.051806 Best loss: 0.034669 Accuracy: 98.91% 37 Validation loss: 0.061325 Best loss: 0.034669 Accuracy: 98.91% 38 Validation loss: 0.049170 Best loss: 0.034669 Accuracy: 98.94% 39 Validation loss: 0.056155 Best loss: 0.034669 Accuracy: 98.79% 40 Validation loss: 0.062011 Best loss: 0.034669 Accuracy: 98.75% 41 Validation loss: 0.047464 Best loss: 0.034669 Accuracy: 99.18% 42 Validation loss: 0.041070 Best loss: 0.034669 Accuracy: 99.22% 43 Validation loss: 0.047821 Best loss: 0.034669 Accuracy: 99.06% 44 Validation loss: 0.046266 Best loss: 0.034669 Accuracy: 98.91% 45 Validation loss: 0.046639 Best loss: 0.034669 Accuracy: 98.98% 46 Validation loss: 0.051553 Best loss: 0.034669 Accuracy: 98.98% 47 Validation loss: 0.060853 Best loss: 0.034669 Accuracy: 98.98% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.8min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01
[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 1.8min remaining: 0.0s
0 Validation loss: 0.137634 Best loss: 0.137634 Accuracy: 96.05% 1 Validation loss: 0.070076 Best loss: 0.070076 Accuracy: 98.01% 2 Validation loss: 0.073236 Best loss: 0.070076 Accuracy: 97.93% 3 Validation loss: 0.067610 Best loss: 0.067610 Accuracy: 98.16% 4 Validation loss: 0.057112 Best loss: 0.057112 Accuracy: 98.63% 5 Validation loss: 0.041931 Best loss: 0.041931 Accuracy: 98.67% 6 Validation loss: 0.043249 Best loss: 0.041931 Accuracy: 98.87% 7 Validation loss: 0.064665 Best loss: 0.041931 Accuracy: 98.12% 8 Validation loss: 0.047341 Best loss: 0.041931 Accuracy: 98.55% 9 Validation loss: 0.052787 Best loss: 0.041931 Accuracy: 98.55% 10 Validation loss: 0.052068 Best loss: 0.041931 Accuracy: 98.59% 11 Validation loss: 0.039048 Best loss: 0.039048 Accuracy: 98.98% 12 Validation loss: 0.038193 Best loss: 0.038193 Accuracy: 99.06% 13 Validation loss: 0.044616 Best loss: 0.038193 Accuracy: 98.98% 14 Validation loss: 0.034230 Best loss: 0.034230 Accuracy: 98.87% 15 Validation loss: 0.061938 Best loss: 0.034230 Accuracy: 98.55% 16 Validation loss: 0.072313 Best loss: 0.034230 Accuracy: 98.51% 17 Validation loss: 0.064909 Best loss: 0.034230 Accuracy: 98.71% 18 Validation loss: 0.045438 Best loss: 0.034230 Accuracy: 98.94% 19 Validation loss: 0.061457 Best loss: 0.034230 Accuracy: 98.67% 20 Validation loss: 0.043851 Best loss: 0.034230 Accuracy: 98.91% 21 Validation loss: 0.048181 Best loss: 0.034230 Accuracy: 98.91% 22 Validation loss: 0.044437 Best loss: 0.034230 Accuracy: 99.18% 23 Validation loss: 0.035096 Best loss: 0.034230 Accuracy: 99.06% 24 Validation loss: 0.040507 Best loss: 0.034230 Accuracy: 99.06% 25 Validation loss: 0.046511 Best loss: 0.034230 Accuracy: 99.10% 26 Validation loss: 0.034422 Best loss: 0.034230 Accuracy: 99.26% 27 Validation loss: 0.041212 Best loss: 0.034230 Accuracy: 98.98% 28 Validation loss: 0.043192 Best loss: 0.034230 Accuracy: 99.06% 29 Validation loss: 0.049234 Best loss: 0.034230 Accuracy: 98.94% 30 Validation loss: 0.039124 Best loss: 0.034230 Accuracy: 99.18% 31 Validation loss: 0.058504 Best loss: 0.034230 Accuracy: 98.79% 32 Validation loss: 0.039917 Best loss: 0.034230 Accuracy: 99.06% 33 Validation loss: 0.033875 Best loss: 0.033875 Accuracy: 99.10% 34 Validation loss: 0.043126 Best loss: 0.033875 Accuracy: 98.91% 35 Validation loss: 0.060313 Best loss: 0.033875 Accuracy: 99.06% 36 Validation loss: 0.048932 Best loss: 0.033875 Accuracy: 99.06% 37 Validation loss: 0.048152 Best loss: 0.033875 Accuracy: 99.10% 38 Validation loss: 0.050425 Best loss: 0.033875 Accuracy: 99.06% 39 Validation loss: 0.049415 Best loss: 0.033875 Accuracy: 98.94% 40 Validation loss: 0.055380 Best loss: 0.033875 Accuracy: 99.10% 41 Validation loss: 0.040577 Best loss: 0.033875 Accuracy: 99.10% 42 Validation loss: 0.044395 Best loss: 0.033875 Accuracy: 99.18% 43 Validation loss: 0.045205 Best loss: 0.033875 Accuracy: 99.34% 44 Validation loss: 0.040960 Best loss: 0.033875 Accuracy: 99.22% 45 Validation loss: 0.035698 Best loss: 0.033875 Accuracy: 99.37% 46 Validation loss: 0.045161 Best loss: 0.033875 Accuracy: 99.22% 47 Validation loss: 0.048006 Best loss: 0.033875 Accuracy: 99.14% 48 Validation loss: 0.052158 Best loss: 0.033875 Accuracy: 99.14% 49 Validation loss: 0.035590 Best loss: 0.033875 Accuracy: 99.30% 50 Validation loss: 0.037913 Best loss: 0.033875 Accuracy: 99.26% 51 Validation loss: 0.044974 Best loss: 0.033875 Accuracy: 99.18% 52 Validation loss: 0.031945 Best loss: 0.031945 Accuracy: 99.37% 53 Validation loss: 0.040864 Best loss: 0.031945 Accuracy: 98.91% 54 Validation loss: 0.039893 Best loss: 0.031945 Accuracy: 99.06% 55 Validation loss: 0.043149 Best loss: 0.031945 Accuracy: 98.91% 56 Validation loss: 0.061883 Best loss: 0.031945 Accuracy: 99.06% 57 Validation loss: 0.051558 Best loss: 0.031945 Accuracy: 98.83% 58 Validation loss: 0.064293 Best loss: 0.031945 Accuracy: 98.79% 59 Validation loss: 0.043205 Best loss: 0.031945 Accuracy: 99.10% 60 Validation loss: 0.035615 Best loss: 0.031945 Accuracy: 99.26% 61 Validation loss: 0.062731 Best loss: 0.031945 Accuracy: 98.91% 62 Validation loss: 0.069868 Best loss: 0.031945 Accuracy: 98.98% 63 Validation loss: 0.084215 Best loss: 0.031945 Accuracy: 98.79% 64 Validation loss: 0.054711 Best loss: 0.031945 Accuracy: 99.06% 65 Validation loss: 0.047209 Best loss: 0.031945 Accuracy: 98.98% 66 Validation loss: 0.042851 Best loss: 0.031945 Accuracy: 99.26% 67 Validation loss: 0.048276 Best loss: 0.031945 Accuracy: 99.26% 68 Validation loss: 0.052396 Best loss: 0.031945 Accuracy: 99.14% 69 Validation loss: 0.064267 Best loss: 0.031945 Accuracy: 99.02% 70 Validation loss: 0.051526 Best loss: 0.031945 Accuracy: 99.02% 71 Validation loss: 0.054529 Best loss: 0.031945 Accuracy: 98.94% 72 Validation loss: 0.047469 Best loss: 0.031945 Accuracy: 98.98% 73 Validation loss: 0.045052 Best loss: 0.031945 Accuracy: 98.98% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 2.8min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.072870 Best loss: 0.072870 Accuracy: 98.01% 1 Validation loss: 0.064845 Best loss: 0.064845 Accuracy: 98.12% 2 Validation loss: 0.060763 Best loss: 0.060763 Accuracy: 97.89% 3 Validation loss: 0.082023 Best loss: 0.060763 Accuracy: 97.11% 4 Validation loss: 0.036360 Best loss: 0.036360 Accuracy: 98.75% 5 Validation loss: 0.049882 Best loss: 0.036360 Accuracy: 98.79% 6 Validation loss: 0.050424 Best loss: 0.036360 Accuracy: 98.75% 7 Validation loss: 0.061546 Best loss: 0.036360 Accuracy: 98.55% 8 Validation loss: 0.053852 Best loss: 0.036360 Accuracy: 98.63% 9 Validation loss: 0.045697 Best loss: 0.036360 Accuracy: 98.79% 10 Validation loss: 0.055674 Best loss: 0.036360 Accuracy: 98.71% 11 Validation loss: 0.046333 Best loss: 0.036360 Accuracy: 98.83% 12 Validation loss: 0.052748 Best loss: 0.036360 Accuracy: 98.87% 13 Validation loss: 0.037892 Best loss: 0.036360 Accuracy: 98.79% 14 Validation loss: 0.045437 Best loss: 0.036360 Accuracy: 98.91% 15 Validation loss: 0.046489 Best loss: 0.036360 Accuracy: 98.79% 16 Validation loss: 0.034489 Best loss: 0.034489 Accuracy: 98.87% 17 Validation loss: 0.038448 Best loss: 0.034489 Accuracy: 98.98% 18 Validation loss: 0.042124 Best loss: 0.034489 Accuracy: 98.94% 19 Validation loss: 0.044723 Best loss: 0.034489 Accuracy: 98.98% 20 Validation loss: 0.025899 Best loss: 0.025899 Accuracy: 99.30% 21 Validation loss: 0.041342 Best loss: 0.025899 Accuracy: 98.98% 22 Validation loss: 0.037453 Best loss: 0.025899 Accuracy: 98.91% 23 Validation loss: 0.030569 Best loss: 0.025899 Accuracy: 99.14% 24 Validation loss: 0.041729 Best loss: 0.025899 Accuracy: 99.06% 25 Validation loss: 0.162361 Best loss: 0.025899 Accuracy: 97.89% 26 Validation loss: 0.055199 Best loss: 0.025899 Accuracy: 98.83% 27 Validation loss: 0.040155 Best loss: 0.025899 Accuracy: 98.91% 28 Validation loss: 0.041570 Best loss: 0.025899 Accuracy: 99.22% 29 Validation loss: 0.048274 Best loss: 0.025899 Accuracy: 98.87% 30 Validation loss: 0.046986 Best loss: 0.025899 Accuracy: 98.79% 31 Validation loss: 0.065739 Best loss: 0.025899 Accuracy: 98.87% 32 Validation loss: 0.035171 Best loss: 0.025899 Accuracy: 99.26% 33 Validation loss: 0.042599 Best loss: 0.025899 Accuracy: 99.26% 34 Validation loss: 0.034077 Best loss: 0.025899 Accuracy: 98.98% 35 Validation loss: 0.036685 Best loss: 0.025899 Accuracy: 99.18% 36 Validation loss: 0.058876 Best loss: 0.025899 Accuracy: 99.02% 37 Validation loss: 0.033328 Best loss: 0.025899 Accuracy: 99.18% 38 Validation loss: 0.058411 Best loss: 0.025899 Accuracy: 98.94% 39 Validation loss: 0.035549 Best loss: 0.025899 Accuracy: 99.26% 40 Validation loss: 0.033373 Best loss: 0.025899 Accuracy: 99.26% 41 Validation loss: 0.047759 Best loss: 0.025899 Accuracy: 99.02% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.6min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.104367 Best loss: 0.104367 Accuracy: 96.79% 1 Validation loss: 0.104119 Best loss: 0.104119 Accuracy: 96.76% 2 Validation loss: 0.065189 Best loss: 0.065189 Accuracy: 97.85% 3 Validation loss: 0.072674 Best loss: 0.065189 Accuracy: 97.42% 4 Validation loss: 0.041901 Best loss: 0.041901 Accuracy: 98.71% 5 Validation loss: 0.055881 Best loss: 0.041901 Accuracy: 98.12% 6 Validation loss: 0.054379 Best loss: 0.041901 Accuracy: 98.55% 7 Validation loss: 0.050530 Best loss: 0.041901 Accuracy: 98.44% 8 Validation loss: 0.052540 Best loss: 0.041901 Accuracy: 98.40% 9 Validation loss: 0.053495 Best loss: 0.041901 Accuracy: 98.59% 10 Validation loss: 0.056523 Best loss: 0.041901 Accuracy: 98.28% 11 Validation loss: 0.049274 Best loss: 0.041901 Accuracy: 98.55% 12 Validation loss: 0.044716 Best loss: 0.041901 Accuracy: 98.79% 13 Validation loss: 0.050429 Best loss: 0.041901 Accuracy: 98.59% 14 Validation loss: 0.046766 Best loss: 0.041901 Accuracy: 98.79% 15 Validation loss: 0.043954 Best loss: 0.041901 Accuracy: 98.63% 16 Validation loss: 0.043191 Best loss: 0.041901 Accuracy: 98.67% 17 Validation loss: 0.038440 Best loss: 0.038440 Accuracy: 99.02% 18 Validation loss: 0.042458 Best loss: 0.038440 Accuracy: 98.87% 19 Validation loss: 0.051925 Best loss: 0.038440 Accuracy: 98.48% 20 Validation loss: 0.056729 Best loss: 0.038440 Accuracy: 98.67% 21 Validation loss: 0.042435 Best loss: 0.038440 Accuracy: 98.94% 22 Validation loss: 0.050565 Best loss: 0.038440 Accuracy: 98.79% 23 Validation loss: 0.039099 Best loss: 0.038440 Accuracy: 98.87% 24 Validation loss: 0.075094 Best loss: 0.038440 Accuracy: 97.62% 25 Validation loss: 0.051802 Best loss: 0.038440 Accuracy: 98.55% 26 Validation loss: 0.039744 Best loss: 0.038440 Accuracy: 98.98% 27 Validation loss: 0.043967 Best loss: 0.038440 Accuracy: 98.83% 28 Validation loss: 0.037986 Best loss: 0.037986 Accuracy: 98.94% 29 Validation loss: 0.063283 Best loss: 0.037986 Accuracy: 98.28% 30 Validation loss: 0.043748 Best loss: 0.037986 Accuracy: 98.79% 31 Validation loss: 0.042245 Best loss: 0.037986 Accuracy: 98.91% 32 Validation loss: 0.037693 Best loss: 0.037693 Accuracy: 99.10% 33 Validation loss: 0.047967 Best loss: 0.037693 Accuracy: 98.79% 34 Validation loss: 0.044779 Best loss: 0.037693 Accuracy: 98.98% 35 Validation loss: 0.067857 Best loss: 0.037693 Accuracy: 98.08% 36 Validation loss: 0.035015 Best loss: 0.035015 Accuracy: 99.06% 37 Validation loss: 0.039891 Best loss: 0.035015 Accuracy: 99.10% 38 Validation loss: 0.040962 Best loss: 0.035015 Accuracy: 98.94% 39 Validation loss: 0.032674 Best loss: 0.032674 Accuracy: 99.18% 40 Validation loss: 0.040850 Best loss: 0.032674 Accuracy: 98.98% 41 Validation loss: 0.048168 Best loss: 0.032674 Accuracy: 98.94% 42 Validation loss: 0.053753 Best loss: 0.032674 Accuracy: 98.48% 43 Validation loss: 0.045564 Best loss: 0.032674 Accuracy: 98.87% 44 Validation loss: 0.043778 Best loss: 0.032674 Accuracy: 98.63% 45 Validation loss: 0.054854 Best loss: 0.032674 Accuracy: 98.94% 46 Validation loss: 0.062135 Best loss: 0.032674 Accuracy: 98.59% 47 Validation loss: 0.078660 Best loss: 0.032674 Accuracy: 98.44% 48 Validation loss: 0.051256 Best loss: 0.032674 Accuracy: 98.91% 49 Validation loss: 0.060288 Best loss: 0.032674 Accuracy: 98.83% 50 Validation loss: 0.043620 Best loss: 0.032674 Accuracy: 98.94% 51 Validation loss: 0.059313 Best loss: 0.032674 Accuracy: 98.28% 52 Validation loss: 0.039889 Best loss: 0.032674 Accuracy: 99.10% 53 Validation loss: 0.041405 Best loss: 0.032674 Accuracy: 99.10% 54 Validation loss: 0.055941 Best loss: 0.032674 Accuracy: 98.67% 55 Validation loss: 0.048463 Best loss: 0.032674 Accuracy: 98.67% 56 Validation loss: 0.046495 Best loss: 0.032674 Accuracy: 98.98% 57 Validation loss: 0.050426 Best loss: 0.032674 Accuracy: 98.87% 58 Validation loss: 0.055754 Best loss: 0.032674 Accuracy: 98.91% 59 Validation loss: 0.058479 Best loss: 0.032674 Accuracy: 98.87% 60 Validation loss: 0.058395 Best loss: 0.032674 Accuracy: 98.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02, total=11.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.112340 Best loss: 0.112340 Accuracy: 96.95% 1 Validation loss: 0.075429 Best loss: 0.075429 Accuracy: 97.81% 2 Validation loss: 0.071592 Best loss: 0.071592 Accuracy: 97.62% 3 Validation loss: 0.046518 Best loss: 0.046518 Accuracy: 98.71% 4 Validation loss: 0.062660 Best loss: 0.046518 Accuracy: 97.97% 5 Validation loss: 0.061883 Best loss: 0.046518 Accuracy: 98.12% 6 Validation loss: 0.050152 Best loss: 0.046518 Accuracy: 98.63% 7 Validation loss: 0.056799 Best loss: 0.046518 Accuracy: 98.44% 8 Validation loss: 0.044233 Best loss: 0.044233 Accuracy: 98.83% 9 Validation loss: 0.058990 Best loss: 0.044233 Accuracy: 98.24% 10 Validation loss: 0.057407 Best loss: 0.044233 Accuracy: 98.40% 11 Validation loss: 0.041414 Best loss: 0.041414 Accuracy: 98.71% 12 Validation loss: 0.045216 Best loss: 0.041414 Accuracy: 98.48% 13 Validation loss: 0.051203 Best loss: 0.041414 Accuracy: 98.40% 14 Validation loss: 0.050323 Best loss: 0.041414 Accuracy: 98.63% 15 Validation loss: 0.045777 Best loss: 0.041414 Accuracy: 98.63% 16 Validation loss: 0.069467 Best loss: 0.041414 Accuracy: 98.08% 17 Validation loss: 0.039310 Best loss: 0.039310 Accuracy: 98.91% 18 Validation loss: 0.053113 Best loss: 0.039310 Accuracy: 98.51% 19 Validation loss: 0.047728 Best loss: 0.039310 Accuracy: 98.75% 20 Validation loss: 0.045555 Best loss: 0.039310 Accuracy: 98.55% 21 Validation loss: 0.036317 Best loss: 0.036317 Accuracy: 98.87% 22 Validation loss: 0.042486 Best loss: 0.036317 Accuracy: 98.79% 23 Validation loss: 0.036937 Best loss: 0.036317 Accuracy: 98.91% 24 Validation loss: 0.049283 Best loss: 0.036317 Accuracy: 98.48% 25 Validation loss: 0.046233 Best loss: 0.036317 Accuracy: 98.55% 26 Validation loss: 0.043307 Best loss: 0.036317 Accuracy: 98.87% 27 Validation loss: 0.056557 Best loss: 0.036317 Accuracy: 98.59% 28 Validation loss: 0.036565 Best loss: 0.036317 Accuracy: 98.98% 29 Validation loss: 0.055914 Best loss: 0.036317 Accuracy: 98.71% 30 Validation loss: 0.047218 Best loss: 0.036317 Accuracy: 98.71% 31 Validation loss: 0.046177 Best loss: 0.036317 Accuracy: 98.67% 32 Validation loss: 0.042972 Best loss: 0.036317 Accuracy: 98.75% 33 Validation loss: 0.035148 Best loss: 0.035148 Accuracy: 98.98% 34 Validation loss: 0.039101 Best loss: 0.035148 Accuracy: 99.10% 35 Validation loss: 0.050256 Best loss: 0.035148 Accuracy: 98.59% 36 Validation loss: 0.053089 Best loss: 0.035148 Accuracy: 98.94% 37 Validation loss: 0.047915 Best loss: 0.035148 Accuracy: 98.87% 38 Validation loss: 0.055334 Best loss: 0.035148 Accuracy: 98.51% 39 Validation loss: 0.042099 Best loss: 0.035148 Accuracy: 98.71% 40 Validation loss: 0.042449 Best loss: 0.035148 Accuracy: 98.71% 41 Validation loss: 0.038841 Best loss: 0.035148 Accuracy: 98.75% 42 Validation loss: 0.055771 Best loss: 0.035148 Accuracy: 98.75% 43 Validation loss: 0.066117 Best loss: 0.035148 Accuracy: 98.24% 44 Validation loss: 0.041269 Best loss: 0.035148 Accuracy: 98.94% 45 Validation loss: 0.055481 Best loss: 0.035148 Accuracy: 98.44% 46 Validation loss: 0.042247 Best loss: 0.035148 Accuracy: 98.98% 47 Validation loss: 0.046872 Best loss: 0.035148 Accuracy: 98.75% 48 Validation loss: 0.043372 Best loss: 0.035148 Accuracy: 98.67% 49 Validation loss: 0.051078 Best loss: 0.035148 Accuracy: 98.83% 50 Validation loss: 0.041636 Best loss: 0.035148 Accuracy: 99.06% 51 Validation loss: 0.045819 Best loss: 0.035148 Accuracy: 98.79% 52 Validation loss: 0.058786 Best loss: 0.035148 Accuracy: 98.83% 53 Validation loss: 0.043479 Best loss: 0.035148 Accuracy: 98.83% 54 Validation loss: 0.076410 Best loss: 0.035148 Accuracy: 98.40% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02, total=10.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.106095 Best loss: 0.106095 Accuracy: 97.07% 1 Validation loss: 0.056068 Best loss: 0.056068 Accuracy: 98.24% 2 Validation loss: 0.067150 Best loss: 0.056068 Accuracy: 97.73% 3 Validation loss: 0.056616 Best loss: 0.056068 Accuracy: 97.97% 4 Validation loss: 0.081417 Best loss: 0.056068 Accuracy: 97.03% 5 Validation loss: 0.049733 Best loss: 0.049733 Accuracy: 98.32% 6 Validation loss: 0.063839 Best loss: 0.049733 Accuracy: 97.81% 7 Validation loss: 0.047131 Best loss: 0.047131 Accuracy: 98.63% 8 Validation loss: 0.037787 Best loss: 0.037787 Accuracy: 98.75% 9 Validation loss: 0.047984 Best loss: 0.037787 Accuracy: 98.32% 10 Validation loss: 0.047828 Best loss: 0.037787 Accuracy: 98.55% 11 Validation loss: 0.043582 Best loss: 0.037787 Accuracy: 98.51% 12 Validation loss: 0.030495 Best loss: 0.030495 Accuracy: 99.06% 13 Validation loss: 0.050945 Best loss: 0.030495 Accuracy: 98.51% 14 Validation loss: 0.049295 Best loss: 0.030495 Accuracy: 98.59% 15 Validation loss: 0.038823 Best loss: 0.030495 Accuracy: 98.79% 16 Validation loss: 0.034033 Best loss: 0.030495 Accuracy: 99.10% 17 Validation loss: 0.045136 Best loss: 0.030495 Accuracy: 98.71% 18 Validation loss: 0.038034 Best loss: 0.030495 Accuracy: 98.67% 19 Validation loss: 0.037448 Best loss: 0.030495 Accuracy: 99.10% 20 Validation loss: 0.039734 Best loss: 0.030495 Accuracy: 98.83% 21 Validation loss: 0.029177 Best loss: 0.029177 Accuracy: 99.18% 22 Validation loss: 0.027611 Best loss: 0.027611 Accuracy: 99.06% 23 Validation loss: 0.066387 Best loss: 0.027611 Accuracy: 98.48% 24 Validation loss: 0.043748 Best loss: 0.027611 Accuracy: 98.83% 25 Validation loss: 0.058160 Best loss: 0.027611 Accuracy: 98.20% 26 Validation loss: 0.032379 Best loss: 0.027611 Accuracy: 99.10% 27 Validation loss: 0.033908 Best loss: 0.027611 Accuracy: 98.98% 28 Validation loss: 0.031673 Best loss: 0.027611 Accuracy: 98.94% 29 Validation loss: 0.047556 Best loss: 0.027611 Accuracy: 98.83% 30 Validation loss: 0.040334 Best loss: 0.027611 Accuracy: 98.87% 31 Validation loss: 0.052219 Best loss: 0.027611 Accuracy: 98.51% 32 Validation loss: 0.033983 Best loss: 0.027611 Accuracy: 98.83% 33 Validation loss: 0.032518 Best loss: 0.027611 Accuracy: 98.94% 34 Validation loss: 0.035802 Best loss: 0.027611 Accuracy: 99.02% 35 Validation loss: 0.035173 Best loss: 0.027611 Accuracy: 99.10% 36 Validation loss: 0.046654 Best loss: 0.027611 Accuracy: 98.44% 37 Validation loss: 0.044848 Best loss: 0.027611 Accuracy: 98.87% 38 Validation loss: 0.037877 Best loss: 0.027611 Accuracy: 98.59% 39 Validation loss: 0.038406 Best loss: 0.027611 Accuracy: 98.94% 40 Validation loss: 0.042961 Best loss: 0.027611 Accuracy: 98.98% 41 Validation loss: 0.039347 Best loss: 0.027611 Accuracy: 98.98% 42 Validation loss: 0.048092 Best loss: 0.027611 Accuracy: 98.83% 43 Validation loss: 0.042175 Best loss: 0.027611 Accuracy: 99.02% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.02, total= 8.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01 0 Validation loss: 0.071818 Best loss: 0.071818 Accuracy: 97.97% 1 Validation loss: 0.068181 Best loss: 0.068181 Accuracy: 98.01% 2 Validation loss: 0.059325 Best loss: 0.059325 Accuracy: 98.28% 3 Validation loss: 0.064259 Best loss: 0.059325 Accuracy: 98.16% 4 Validation loss: 0.050640 Best loss: 0.050640 Accuracy: 98.44% 5 Validation loss: 0.051917 Best loss: 0.050640 Accuracy: 98.28% 6 Validation loss: 0.045705 Best loss: 0.045705 Accuracy: 98.67% 7 Validation loss: 0.053764 Best loss: 0.045705 Accuracy: 98.40% 8 Validation loss: 0.049099 Best loss: 0.045705 Accuracy: 98.71% 9 Validation loss: 0.046137 Best loss: 0.045705 Accuracy: 98.67% 10 Validation loss: 0.053786 Best loss: 0.045705 Accuracy: 98.75% 11 Validation loss: 0.048873 Best loss: 0.045705 Accuracy: 98.71% 12 Validation loss: 0.060135 Best loss: 0.045705 Accuracy: 98.36% 13 Validation loss: 0.049416 Best loss: 0.045705 Accuracy: 98.71% 14 Validation loss: 0.043894 Best loss: 0.043894 Accuracy: 98.87% 15 Validation loss: 0.062327 Best loss: 0.043894 Accuracy: 98.51% 16 Validation loss: 0.056842 Best loss: 0.043894 Accuracy: 98.67% 17 Validation loss: 0.051270 Best loss: 0.043894 Accuracy: 98.79% 18 Validation loss: 0.074579 Best loss: 0.043894 Accuracy: 98.20% 19 Validation loss: 0.060488 Best loss: 0.043894 Accuracy: 98.44% 20 Validation loss: 0.068257 Best loss: 0.043894 Accuracy: 98.63% 21 Validation loss: 0.051789 Best loss: 0.043894 Accuracy: 98.67% 22 Validation loss: 0.062104 Best loss: 0.043894 Accuracy: 98.55% 23 Validation loss: 0.044263 Best loss: 0.043894 Accuracy: 98.79% 24 Validation loss: 0.046451 Best loss: 0.043894 Accuracy: 98.94% 25 Validation loss: 0.050836 Best loss: 0.043894 Accuracy: 98.91% 26 Validation loss: 0.052117 Best loss: 0.043894 Accuracy: 98.94% 27 Validation loss: 0.055685 Best loss: 0.043894 Accuracy: 98.75% 28 Validation loss: 0.056597 Best loss: 0.043894 Accuracy: 98.91% 29 Validation loss: 0.051686 Best loss: 0.043894 Accuracy: 98.91% 30 Validation loss: 0.049539 Best loss: 0.043894 Accuracy: 98.87% 31 Validation loss: 0.065307 Best loss: 0.043894 Accuracy: 98.63% 32 Validation loss: 0.054515 Best loss: 0.043894 Accuracy: 98.87% 33 Validation loss: 0.052277 Best loss: 0.043894 Accuracy: 98.79% 34 Validation loss: 0.053220 Best loss: 0.043894 Accuracy: 98.63% 35 Validation loss: 0.052381 Best loss: 0.043894 Accuracy: 98.94% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01, total= 51.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01 0 Validation loss: 0.093822 Best loss: 0.093822 Accuracy: 97.07% 1 Validation loss: 0.056536 Best loss: 0.056536 Accuracy: 98.40% 2 Validation loss: 0.049459 Best loss: 0.049459 Accuracy: 98.36% 3 Validation loss: 0.048266 Best loss: 0.048266 Accuracy: 98.59% 4 Validation loss: 0.067281 Best loss: 0.048266 Accuracy: 97.81% 5 Validation loss: 0.043144 Best loss: 0.043144 Accuracy: 98.83% 6 Validation loss: 0.042572 Best loss: 0.042572 Accuracy: 98.79% 7 Validation loss: 0.055343 Best loss: 0.042572 Accuracy: 98.44% 8 Validation loss: 0.047964 Best loss: 0.042572 Accuracy: 98.71% 9 Validation loss: 0.088831 Best loss: 0.042572 Accuracy: 97.73% 10 Validation loss: 0.055745 Best loss: 0.042572 Accuracy: 98.32% 11 Validation loss: 0.054123 Best loss: 0.042572 Accuracy: 98.51% 12 Validation loss: 0.044206 Best loss: 0.042572 Accuracy: 98.63% 13 Validation loss: 0.056324 Best loss: 0.042572 Accuracy: 97.97% 14 Validation loss: 0.048610 Best loss: 0.042572 Accuracy: 98.55% 15 Validation loss: 0.051247 Best loss: 0.042572 Accuracy: 98.63% 16 Validation loss: 0.057649 Best loss: 0.042572 Accuracy: 98.55% 17 Validation loss: 0.068375 Best loss: 0.042572 Accuracy: 98.44% 18 Validation loss: 0.046981 Best loss: 0.042572 Accuracy: 98.75% 19 Validation loss: 0.049126 Best loss: 0.042572 Accuracy: 98.98% 20 Validation loss: 0.047929 Best loss: 0.042572 Accuracy: 98.91% 21 Validation loss: 0.056546 Best loss: 0.042572 Accuracy: 98.75% 22 Validation loss: 0.053283 Best loss: 0.042572 Accuracy: 98.67% 23 Validation loss: 0.068199 Best loss: 0.042572 Accuracy: 98.24% 24 Validation loss: 0.050027 Best loss: 0.042572 Accuracy: 98.87% 25 Validation loss: 0.043331 Best loss: 0.042572 Accuracy: 98.87% 26 Validation loss: 0.050805 Best loss: 0.042572 Accuracy: 98.75% 27 Validation loss: 0.058207 Best loss: 0.042572 Accuracy: 98.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01, total= 40.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01 0 Validation loss: 0.077713 Best loss: 0.077713 Accuracy: 97.62% 1 Validation loss: 0.063839 Best loss: 0.063839 Accuracy: 97.97% 2 Validation loss: 0.053239 Best loss: 0.053239 Accuracy: 98.32% 3 Validation loss: 0.049002 Best loss: 0.049002 Accuracy: 98.63% 4 Validation loss: 0.043206 Best loss: 0.043206 Accuracy: 98.71% 5 Validation loss: 0.035472 Best loss: 0.035472 Accuracy: 98.71% 6 Validation loss: 0.038703 Best loss: 0.035472 Accuracy: 98.79% 7 Validation loss: 0.038140 Best loss: 0.035472 Accuracy: 98.83% 8 Validation loss: 0.053382 Best loss: 0.035472 Accuracy: 98.51% 9 Validation loss: 0.036055 Best loss: 0.035472 Accuracy: 98.91% 10 Validation loss: 0.060132 Best loss: 0.035472 Accuracy: 98.32% 11 Validation loss: 0.045723 Best loss: 0.035472 Accuracy: 98.83% 12 Validation loss: 0.040204 Best loss: 0.035472 Accuracy: 98.91% 13 Validation loss: 0.044850 Best loss: 0.035472 Accuracy: 98.71% 14 Validation loss: 0.059381 Best loss: 0.035472 Accuracy: 98.59% 15 Validation loss: 0.059510 Best loss: 0.035472 Accuracy: 98.48% 16 Validation loss: 0.041853 Best loss: 0.035472 Accuracy: 98.59% 17 Validation loss: 0.042206 Best loss: 0.035472 Accuracy: 98.91% 18 Validation loss: 0.046199 Best loss: 0.035472 Accuracy: 98.83% 19 Validation loss: 0.041314 Best loss: 0.035472 Accuracy: 98.91% 20 Validation loss: 0.055666 Best loss: 0.035472 Accuracy: 98.71% 21 Validation loss: 0.047131 Best loss: 0.035472 Accuracy: 98.94% 22 Validation loss: 0.054525 Best loss: 0.035472 Accuracy: 98.75% 23 Validation loss: 0.051681 Best loss: 0.035472 Accuracy: 98.71% 24 Validation loss: 0.062585 Best loss: 0.035472 Accuracy: 98.67% 25 Validation loss: 0.062679 Best loss: 0.035472 Accuracy: 98.67% 26 Validation loss: 0.056928 Best loss: 0.035472 Accuracy: 98.67% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=30, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.01, total= 38.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.091725 Best loss: 0.091725 Accuracy: 97.30% 1 Validation loss: 0.099285 Best loss: 0.091725 Accuracy: 96.60% 2 Validation loss: 0.056636 Best loss: 0.056636 Accuracy: 98.32% 3 Validation loss: 0.051585 Best loss: 0.051585 Accuracy: 98.44% 4 Validation loss: 0.061245 Best loss: 0.051585 Accuracy: 98.71% 5 Validation loss: 0.060288 Best loss: 0.051585 Accuracy: 98.24% 6 Validation loss: 0.046271 Best loss: 0.046271 Accuracy: 98.59% 7 Validation loss: 0.040220 Best loss: 0.040220 Accuracy: 98.79% 8 Validation loss: 0.042125 Best loss: 0.040220 Accuracy: 98.75% 9 Validation loss: 0.038450 Best loss: 0.038450 Accuracy: 98.71% 10 Validation loss: 0.059078 Best loss: 0.038450 Accuracy: 98.36% 11 Validation loss: 0.060027 Best loss: 0.038450 Accuracy: 97.97% 12 Validation loss: 0.040285 Best loss: 0.038450 Accuracy: 98.91% 13 Validation loss: 0.039818 Best loss: 0.038450 Accuracy: 98.83% 14 Validation loss: 0.041295 Best loss: 0.038450 Accuracy: 98.63% 15 Validation loss: 0.036652 Best loss: 0.036652 Accuracy: 98.83% 16 Validation loss: 0.046955 Best loss: 0.036652 Accuracy: 98.83% 17 Validation loss: 0.037620 Best loss: 0.036652 Accuracy: 98.91% 18 Validation loss: 0.038915 Best loss: 0.036652 Accuracy: 98.98% 19 Validation loss: 0.052817 Best loss: 0.036652 Accuracy: 98.40% 20 Validation loss: 0.055311 Best loss: 0.036652 Accuracy: 98.79% 21 Validation loss: 0.041558 Best loss: 0.036652 Accuracy: 99.02% 22 Validation loss: 0.050212 Best loss: 0.036652 Accuracy: 98.91% 23 Validation loss: 0.040169 Best loss: 0.036652 Accuracy: 99.10% 24 Validation loss: 0.043748 Best loss: 0.036652 Accuracy: 98.55% 25 Validation loss: 0.035377 Best loss: 0.035377 Accuracy: 99.06% 26 Validation loss: 0.042383 Best loss: 0.035377 Accuracy: 99.06% 27 Validation loss: 0.054164 Best loss: 0.035377 Accuracy: 98.59% 28 Validation loss: 0.050271 Best loss: 0.035377 Accuracy: 98.94% 29 Validation loss: 0.040496 Best loss: 0.035377 Accuracy: 98.94% 30 Validation loss: 0.047016 Best loss: 0.035377 Accuracy: 98.87% 31 Validation loss: 0.046612 Best loss: 0.035377 Accuracy: 99.02% 32 Validation loss: 0.038143 Best loss: 0.035377 Accuracy: 99.10% 33 Validation loss: 0.039829 Best loss: 0.035377 Accuracy: 98.91% 34 Validation loss: 0.054967 Best loss: 0.035377 Accuracy: 98.98% 35 Validation loss: 0.064483 Best loss: 0.035377 Accuracy: 98.48% 36 Validation loss: 0.042554 Best loss: 0.035377 Accuracy: 98.94% 37 Validation loss: 0.049560 Best loss: 0.035377 Accuracy: 98.91% 38 Validation loss: 0.048690 Best loss: 0.035377 Accuracy: 99.14% 39 Validation loss: 0.036430 Best loss: 0.035377 Accuracy: 99.10% 40 Validation loss: 0.046883 Best loss: 0.035377 Accuracy: 99.06% 41 Validation loss: 0.035597 Best loss: 0.035377 Accuracy: 99.02% 42 Validation loss: 0.045876 Best loss: 0.035377 Accuracy: 98.75% 43 Validation loss: 0.048113 Best loss: 0.035377 Accuracy: 98.83% 44 Validation loss: 0.044196 Best loss: 0.035377 Accuracy: 98.79% 45 Validation loss: 0.043503 Best loss: 0.035377 Accuracy: 98.83% 46 Validation loss: 0.052241 Best loss: 0.035377 Accuracy: 98.71% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02, total= 9.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.120647 Best loss: 0.120647 Accuracy: 96.29% 1 Validation loss: 0.072047 Best loss: 0.072047 Accuracy: 98.08% 2 Validation loss: 0.056499 Best loss: 0.056499 Accuracy: 98.44% 3 Validation loss: 0.062044 Best loss: 0.056499 Accuracy: 97.97% 4 Validation loss: 0.053203 Best loss: 0.053203 Accuracy: 98.40% 5 Validation loss: 0.045972 Best loss: 0.045972 Accuracy: 98.51% 6 Validation loss: 0.052936 Best loss: 0.045972 Accuracy: 98.63% 7 Validation loss: 0.054882 Best loss: 0.045972 Accuracy: 98.20% 8 Validation loss: 0.038410 Best loss: 0.038410 Accuracy: 98.98% 9 Validation loss: 0.049357 Best loss: 0.038410 Accuracy: 98.55% 10 Validation loss: 0.041072 Best loss: 0.038410 Accuracy: 98.75% 11 Validation loss: 0.046495 Best loss: 0.038410 Accuracy: 98.55% 12 Validation loss: 0.038146 Best loss: 0.038146 Accuracy: 98.91% 13 Validation loss: 0.037672 Best loss: 0.037672 Accuracy: 98.79% 14 Validation loss: 0.048455 Best loss: 0.037672 Accuracy: 98.75% 15 Validation loss: 0.049075 Best loss: 0.037672 Accuracy: 98.63% 16 Validation loss: 0.108195 Best loss: 0.037672 Accuracy: 97.34% 17 Validation loss: 0.040059 Best loss: 0.037672 Accuracy: 98.55% 18 Validation loss: 0.036603 Best loss: 0.036603 Accuracy: 98.87% 19 Validation loss: 0.047125 Best loss: 0.036603 Accuracy: 98.91% 20 Validation loss: 0.034016 Best loss: 0.034016 Accuracy: 99.02% 21 Validation loss: 0.048498 Best loss: 0.034016 Accuracy: 98.83% 22 Validation loss: 0.038915 Best loss: 0.034016 Accuracy: 98.83% 23 Validation loss: 0.040321 Best loss: 0.034016 Accuracy: 99.06% 24 Validation loss: 0.036787 Best loss: 0.034016 Accuracy: 98.87% 25 Validation loss: 0.032589 Best loss: 0.032589 Accuracy: 99.06% 26 Validation loss: 0.036127 Best loss: 0.032589 Accuracy: 98.87% 27 Validation loss: 0.040345 Best loss: 0.032589 Accuracy: 98.67% 28 Validation loss: 0.046692 Best loss: 0.032589 Accuracy: 98.75% 29 Validation loss: 0.047825 Best loss: 0.032589 Accuracy: 98.75% 30 Validation loss: 0.047240 Best loss: 0.032589 Accuracy: 98.98% 31 Validation loss: 0.054593 Best loss: 0.032589 Accuracy: 98.67% 32 Validation loss: 0.044374 Best loss: 0.032589 Accuracy: 98.94% 33 Validation loss: 0.054006 Best loss: 0.032589 Accuracy: 98.59% 34 Validation loss: 0.033791 Best loss: 0.032589 Accuracy: 98.94% 35 Validation loss: 0.037733 Best loss: 0.032589 Accuracy: 98.79% 36 Validation loss: 0.047852 Best loss: 0.032589 Accuracy: 99.10% 37 Validation loss: 0.038547 Best loss: 0.032589 Accuracy: 98.87% 38 Validation loss: 0.047430 Best loss: 0.032589 Accuracy: 98.94% 39 Validation loss: 0.039120 Best loss: 0.032589 Accuracy: 99.06% 40 Validation loss: 0.036384 Best loss: 0.032589 Accuracy: 99.22% 41 Validation loss: 0.040186 Best loss: 0.032589 Accuracy: 99.06% 42 Validation loss: 0.055761 Best loss: 0.032589 Accuracy: 98.83% 43 Validation loss: 0.096384 Best loss: 0.032589 Accuracy: 98.71% 44 Validation loss: 0.039013 Best loss: 0.032589 Accuracy: 98.94% 45 Validation loss: 0.074751 Best loss: 0.032589 Accuracy: 98.36% 46 Validation loss: 0.044340 Best loss: 0.032589 Accuracy: 99.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02, total= 8.8min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.090181 Best loss: 0.090181 Accuracy: 97.62% 1 Validation loss: 0.058239 Best loss: 0.058239 Accuracy: 98.28% 2 Validation loss: 0.093805 Best loss: 0.058239 Accuracy: 96.91% 3 Validation loss: 0.048987 Best loss: 0.048987 Accuracy: 98.67% 4 Validation loss: 0.060964 Best loss: 0.048987 Accuracy: 98.05% 5 Validation loss: 0.045847 Best loss: 0.045847 Accuracy: 98.40% 6 Validation loss: 0.050831 Best loss: 0.045847 Accuracy: 98.40% 7 Validation loss: 0.036900 Best loss: 0.036900 Accuracy: 98.79% 8 Validation loss: 0.040435 Best loss: 0.036900 Accuracy: 98.71% 9 Validation loss: 0.040293 Best loss: 0.036900 Accuracy: 98.87% 10 Validation loss: 0.041293 Best loss: 0.036900 Accuracy: 98.59% 11 Validation loss: 0.047752 Best loss: 0.036900 Accuracy: 98.63% 12 Validation loss: 0.034073 Best loss: 0.034073 Accuracy: 99.06% 13 Validation loss: 0.033711 Best loss: 0.033711 Accuracy: 98.94% 14 Validation loss: 0.040351 Best loss: 0.033711 Accuracy: 98.67% 15 Validation loss: 0.034676 Best loss: 0.033711 Accuracy: 98.91% 16 Validation loss: 0.041263 Best loss: 0.033711 Accuracy: 98.75% 17 Validation loss: 0.037964 Best loss: 0.033711 Accuracy: 98.71% 18 Validation loss: 0.030531 Best loss: 0.030531 Accuracy: 99.14% 19 Validation loss: 0.028382 Best loss: 0.028382 Accuracy: 99.10% 20 Validation loss: 0.044871 Best loss: 0.028382 Accuracy: 98.71% 21 Validation loss: 0.028888 Best loss: 0.028382 Accuracy: 99.10% 22 Validation loss: 0.032997 Best loss: 0.028382 Accuracy: 98.98% 23 Validation loss: 0.036426 Best loss: 0.028382 Accuracy: 99.22% 24 Validation loss: 0.042457 Best loss: 0.028382 Accuracy: 98.83% 25 Validation loss: 0.039613 Best loss: 0.028382 Accuracy: 98.75% 26 Validation loss: 0.029097 Best loss: 0.028382 Accuracy: 99.26% 27 Validation loss: 0.034804 Best loss: 0.028382 Accuracy: 98.83% 28 Validation loss: 0.027794 Best loss: 0.027794 Accuracy: 99.14% 29 Validation loss: 0.029787 Best loss: 0.027794 Accuracy: 98.91% 30 Validation loss: 0.066690 Best loss: 0.027794 Accuracy: 98.08% 31 Validation loss: 0.042921 Best loss: 0.027794 Accuracy: 98.67% 32 Validation loss: 0.033850 Best loss: 0.027794 Accuracy: 98.94% 33 Validation loss: 0.038670 Best loss: 0.027794 Accuracy: 98.91% 34 Validation loss: 0.032448 Best loss: 0.027794 Accuracy: 99.10% 35 Validation loss: 0.038640 Best loss: 0.027794 Accuracy: 98.91% 36 Validation loss: 0.032984 Best loss: 0.027794 Accuracy: 99.14% 37 Validation loss: 0.054012 Best loss: 0.027794 Accuracy: 98.75% 38 Validation loss: 0.034994 Best loss: 0.027794 Accuracy: 99.14% 39 Validation loss: 0.039777 Best loss: 0.027794 Accuracy: 99.02% 40 Validation loss: 0.042471 Best loss: 0.027794 Accuracy: 98.87% 41 Validation loss: 0.040195 Best loss: 0.027794 Accuracy: 99.06% 42 Validation loss: 0.057741 Best loss: 0.027794 Accuracy: 98.55% 43 Validation loss: 0.026017 Best loss: 0.026017 Accuracy: 99.22% 44 Validation loss: 0.034930 Best loss: 0.026017 Accuracy: 99.18% 45 Validation loss: 0.050024 Best loss: 0.026017 Accuracy: 99.02% 46 Validation loss: 0.047276 Best loss: 0.026017 Accuracy: 98.94% 47 Validation loss: 0.047496 Best loss: 0.026017 Accuracy: 98.75% 48 Validation loss: 0.039223 Best loss: 0.026017 Accuracy: 99.18% 49 Validation loss: 0.068178 Best loss: 0.026017 Accuracy: 98.87% 50 Validation loss: 0.044304 Best loss: 0.026017 Accuracy: 98.91% 51 Validation loss: 0.057439 Best loss: 0.026017 Accuracy: 98.63% 52 Validation loss: 0.052894 Best loss: 0.026017 Accuracy: 98.71% 53 Validation loss: 0.030465 Best loss: 0.026017 Accuracy: 99.14% 54 Validation loss: 0.070329 Best loss: 0.026017 Accuracy: 98.40% 55 Validation loss: 0.059983 Best loss: 0.026017 Accuracy: 98.75% 56 Validation loss: 0.045243 Best loss: 0.026017 Accuracy: 98.91% 57 Validation loss: 0.038966 Best loss: 0.026017 Accuracy: 99.10% 58 Validation loss: 0.049804 Best loss: 0.026017 Accuracy: 98.91% 59 Validation loss: 0.059277 Best loss: 0.026017 Accuracy: 98.91% 60 Validation loss: 0.042634 Best loss: 0.026017 Accuracy: 99.02% 61 Validation loss: 0.038932 Best loss: 0.026017 Accuracy: 99.10% 62 Validation loss: 0.039107 Best loss: 0.026017 Accuracy: 99.10% 63 Validation loss: 0.062110 Best loss: 0.026017 Accuracy: 99.06% 64 Validation loss: 0.031779 Best loss: 0.026017 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=10, batch_norm_momentum=0.95, learning_rate=0.02, total=12.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.087340 Best loss: 0.087340 Accuracy: 97.38% 1 Validation loss: 0.083468 Best loss: 0.083468 Accuracy: 97.19% 2 Validation loss: 0.090257 Best loss: 0.083468 Accuracy: 97.34% 3 Validation loss: 0.056239 Best loss: 0.056239 Accuracy: 98.36% 4 Validation loss: 0.071096 Best loss: 0.056239 Accuracy: 97.97% 5 Validation loss: 0.058409 Best loss: 0.056239 Accuracy: 98.24% 6 Validation loss: 0.051663 Best loss: 0.051663 Accuracy: 98.12% 7 Validation loss: 0.058092 Best loss: 0.051663 Accuracy: 98.55% 8 Validation loss: 0.119673 Best loss: 0.051663 Accuracy: 97.58% 9 Validation loss: 0.070695 Best loss: 0.051663 Accuracy: 98.24% 10 Validation loss: 0.068173 Best loss: 0.051663 Accuracy: 98.32% 11 Validation loss: 0.133848 Best loss: 0.051663 Accuracy: 97.26% 12 Validation loss: 0.042312 Best loss: 0.042312 Accuracy: 98.79% 13 Validation loss: 0.056180 Best loss: 0.042312 Accuracy: 98.40% 14 Validation loss: 0.065741 Best loss: 0.042312 Accuracy: 98.40% 15 Validation loss: 0.055282 Best loss: 0.042312 Accuracy: 98.55% 16 Validation loss: 0.073096 Best loss: 0.042312 Accuracy: 98.48% 17 Validation loss: 0.070713 Best loss: 0.042312 Accuracy: 98.48% 18 Validation loss: 0.068235 Best loss: 0.042312 Accuracy: 98.40% 19 Validation loss: 0.052438 Best loss: 0.042312 Accuracy: 98.79% 20 Validation loss: 0.059756 Best loss: 0.042312 Accuracy: 98.59% 21 Validation loss: 0.057647 Best loss: 0.042312 Accuracy: 98.94% 22 Validation loss: 0.076521 Best loss: 0.042312 Accuracy: 98.01% 23 Validation loss: 0.058426 Best loss: 0.042312 Accuracy: 98.71% 24 Validation loss: 0.072542 Best loss: 0.042312 Accuracy: 98.83% 25 Validation loss: 0.071322 Best loss: 0.042312 Accuracy: 98.63% 26 Validation loss: 0.092394 Best loss: 0.042312 Accuracy: 98.55% 27 Validation loss: 0.100242 Best loss: 0.042312 Accuracy: 97.89% 28 Validation loss: 0.063593 Best loss: 0.042312 Accuracy: 98.63% 29 Validation loss: 0.086675 Best loss: 0.042312 Accuracy: 98.67% 30 Validation loss: 0.053535 Best loss: 0.042312 Accuracy: 98.94% 31 Validation loss: 0.067996 Best loss: 0.042312 Accuracy: 98.79% 32 Validation loss: 0.058059 Best loss: 0.042312 Accuracy: 98.71% 33 Validation loss: 0.051385 Best loss: 0.042312 Accuracy: 99.02% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02, total= 42.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.181523 Best loss: 0.181523 Accuracy: 94.96% 1 Validation loss: 0.064672 Best loss: 0.064672 Accuracy: 97.77% 2 Validation loss: 0.059800 Best loss: 0.059800 Accuracy: 98.24% 3 Validation loss: 0.078875 Best loss: 0.059800 Accuracy: 97.93% 4 Validation loss: 0.091761 Best loss: 0.059800 Accuracy: 97.30% 5 Validation loss: 0.053515 Best loss: 0.053515 Accuracy: 98.63% 6 Validation loss: 0.075468 Best loss: 0.053515 Accuracy: 98.12% 7 Validation loss: 0.049923 Best loss: 0.049923 Accuracy: 98.59% 8 Validation loss: 0.056249 Best loss: 0.049923 Accuracy: 98.63% 9 Validation loss: 0.069097 Best loss: 0.049923 Accuracy: 98.63% 10 Validation loss: 0.040973 Best loss: 0.040973 Accuracy: 98.79% 11 Validation loss: 0.085927 Best loss: 0.040973 Accuracy: 98.08% 12 Validation loss: 0.089379 Best loss: 0.040973 Accuracy: 98.32% 13 Validation loss: 0.057850 Best loss: 0.040973 Accuracy: 98.83% 14 Validation loss: 0.053549 Best loss: 0.040973 Accuracy: 98.71% 15 Validation loss: 0.052837 Best loss: 0.040973 Accuracy: 98.71% 16 Validation loss: 0.053639 Best loss: 0.040973 Accuracy: 98.94% 17 Validation loss: 0.072089 Best loss: 0.040973 Accuracy: 98.59% 18 Validation loss: 0.054469 Best loss: 0.040973 Accuracy: 98.67% 19 Validation loss: 0.051603 Best loss: 0.040973 Accuracy: 98.79% 20 Validation loss: 0.112685 Best loss: 0.040973 Accuracy: 98.12% 21 Validation loss: 0.048639 Best loss: 0.040973 Accuracy: 98.79% 22 Validation loss: 0.063688 Best loss: 0.040973 Accuracy: 98.28% 23 Validation loss: 0.080090 Best loss: 0.040973 Accuracy: 98.71% 24 Validation loss: 0.053289 Best loss: 0.040973 Accuracy: 98.94% 25 Validation loss: 0.050911 Best loss: 0.040973 Accuracy: 98.79% 26 Validation loss: 0.073259 Best loss: 0.040973 Accuracy: 98.44% 27 Validation loss: 0.066756 Best loss: 0.040973 Accuracy: 98.79% 28 Validation loss: 0.075304 Best loss: 0.040973 Accuracy: 98.48% 29 Validation loss: 0.071768 Best loss: 0.040973 Accuracy: 98.71% 30 Validation loss: 0.077596 Best loss: 0.040973 Accuracy: 98.51% 31 Validation loss: 0.068593 Best loss: 0.040973 Accuracy: 98.71% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02, total= 41.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.143712 Best loss: 0.143712 Accuracy: 96.21% 1 Validation loss: 0.068026 Best loss: 0.068026 Accuracy: 98.01% 2 Validation loss: 0.066922 Best loss: 0.066922 Accuracy: 97.85% 3 Validation loss: 0.066971 Best loss: 0.066922 Accuracy: 97.81% 4 Validation loss: 0.060470 Best loss: 0.060470 Accuracy: 98.36% 5 Validation loss: 0.032672 Best loss: 0.032672 Accuracy: 98.91% 6 Validation loss: 0.054736 Best loss: 0.032672 Accuracy: 98.63% 7 Validation loss: 0.063604 Best loss: 0.032672 Accuracy: 98.59% 8 Validation loss: 0.053026 Best loss: 0.032672 Accuracy: 98.67% 9 Validation loss: 0.053880 Best loss: 0.032672 Accuracy: 98.48% 10 Validation loss: 0.053668 Best loss: 0.032672 Accuracy: 98.91% 11 Validation loss: 0.066549 Best loss: 0.032672 Accuracy: 98.44% 12 Validation loss: 0.051400 Best loss: 0.032672 Accuracy: 98.79% 13 Validation loss: 0.105830 Best loss: 0.032672 Accuracy: 97.89% 14 Validation loss: 0.066711 Best loss: 0.032672 Accuracy: 98.63% 15 Validation loss: 0.053379 Best loss: 0.032672 Accuracy: 98.59% 16 Validation loss: 0.071849 Best loss: 0.032672 Accuracy: 98.67% 17 Validation loss: 0.057810 Best loss: 0.032672 Accuracy: 98.67% 18 Validation loss: 0.086824 Best loss: 0.032672 Accuracy: 98.01% 19 Validation loss: 0.051554 Best loss: 0.032672 Accuracy: 98.71% 20 Validation loss: 0.063032 Best loss: 0.032672 Accuracy: 98.55% 21 Validation loss: 0.046818 Best loss: 0.032672 Accuracy: 98.87% 22 Validation loss: 0.052484 Best loss: 0.032672 Accuracy: 98.94% 23 Validation loss: 0.040887 Best loss: 0.032672 Accuracy: 98.98% 24 Validation loss: 0.060387 Best loss: 0.032672 Accuracy: 98.83% 25 Validation loss: 0.050275 Best loss: 0.032672 Accuracy: 99.06% 26 Validation loss: 0.067204 Best loss: 0.032672 Accuracy: 98.75% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.02, total= 34.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.185905 Best loss: 0.185905 Accuracy: 95.66% 1 Validation loss: 0.104295 Best loss: 0.104295 Accuracy: 97.38% 2 Validation loss: 0.161749 Best loss: 0.104295 Accuracy: 95.54% 3 Validation loss: 0.089598 Best loss: 0.089598 Accuracy: 97.42% 4 Validation loss: 0.287289 Best loss: 0.089598 Accuracy: 95.70% 5 Validation loss: 0.055920 Best loss: 0.055920 Accuracy: 98.59% 6 Validation loss: 0.054260 Best loss: 0.054260 Accuracy: 98.63% 7 Validation loss: 0.092470 Best loss: 0.054260 Accuracy: 98.55% 8 Validation loss: 0.073550 Best loss: 0.054260 Accuracy: 98.55% 9 Validation loss: 0.101270 Best loss: 0.054260 Accuracy: 98.01% 10 Validation loss: 0.055455 Best loss: 0.054260 Accuracy: 98.63% 11 Validation loss: 0.075049 Best loss: 0.054260 Accuracy: 98.63% 12 Validation loss: 0.140685 Best loss: 0.054260 Accuracy: 98.12% 13 Validation loss: 0.203977 Best loss: 0.054260 Accuracy: 97.97% 14 Validation loss: 0.213019 Best loss: 0.054260 Accuracy: 98.12% 15 Validation loss: 0.758607 Best loss: 0.054260 Accuracy: 96.68% 16 Validation loss: 0.080875 Best loss: 0.054260 Accuracy: 98.94% 17 Validation loss: 0.078833 Best loss: 0.054260 Accuracy: 98.91% 18 Validation loss: 0.062131 Best loss: 0.054260 Accuracy: 99.10% 19 Validation loss: 0.098332 Best loss: 0.054260 Accuracy: 98.75% 20 Validation loss: 0.069427 Best loss: 0.054260 Accuracy: 98.87% 21 Validation loss: 0.112902 Best loss: 0.054260 Accuracy: 98.75% 22 Validation loss: 0.115140 Best loss: 0.054260 Accuracy: 98.98% 23 Validation loss: 0.142818 Best loss: 0.054260 Accuracy: 98.05% 24 Validation loss: 0.430688 Best loss: 0.054260 Accuracy: 97.93% 25 Validation loss: 0.525711 Best loss: 0.054260 Accuracy: 98.05% 26 Validation loss: 0.136074 Best loss: 0.054260 Accuracy: 98.63% 27 Validation loss: 0.105667 Best loss: 0.054260 Accuracy: 98.71% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 37.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.141659 Best loss: 0.141659 Accuracy: 96.48% 1 Validation loss: 0.091402 Best loss: 0.091402 Accuracy: 97.69% 2 Validation loss: 0.089777 Best loss: 0.089777 Accuracy: 97.65% 3 Validation loss: 0.191061 Best loss: 0.089777 Accuracy: 95.82% 4 Validation loss: 0.175135 Best loss: 0.089777 Accuracy: 96.87% 5 Validation loss: 0.054552 Best loss: 0.054552 Accuracy: 98.59% 6 Validation loss: 0.073306 Best loss: 0.054552 Accuracy: 97.93% 7 Validation loss: 0.112561 Best loss: 0.054552 Accuracy: 97.22% 8 Validation loss: 0.202782 Best loss: 0.054552 Accuracy: 97.50% 9 Validation loss: 0.088841 Best loss: 0.054552 Accuracy: 98.44% 10 Validation loss: 0.076430 Best loss: 0.054552 Accuracy: 98.32% 11 Validation loss: 0.135920 Best loss: 0.054552 Accuracy: 98.01% 12 Validation loss: 0.085372 Best loss: 0.054552 Accuracy: 98.87% 13 Validation loss: 0.166792 Best loss: 0.054552 Accuracy: 98.20% 14 Validation loss: 0.056816 Best loss: 0.054552 Accuracy: 98.67% 15 Validation loss: 0.059081 Best loss: 0.054552 Accuracy: 98.79% 16 Validation loss: 0.052497 Best loss: 0.052497 Accuracy: 98.94% 17 Validation loss: 0.105631 Best loss: 0.052497 Accuracy: 98.20% 18 Validation loss: 0.090692 Best loss: 0.052497 Accuracy: 98.55% 19 Validation loss: 0.092306 Best loss: 0.052497 Accuracy: 98.83% 20 Validation loss: 107.878105 Best loss: 0.052497 Accuracy: 84.75% 21 Validation loss: 0.678226 Best loss: 0.052497 Accuracy: 96.99% 22 Validation loss: 0.199745 Best loss: 0.052497 Accuracy: 96.83% 23 Validation loss: 0.124675 Best loss: 0.052497 Accuracy: 98.98% 24 Validation loss: 0.134944 Best loss: 0.052497 Accuracy: 98.98% 25 Validation loss: 0.128509 Best loss: 0.052497 Accuracy: 98.71% 26 Validation loss: 0.149788 Best loss: 0.052497 Accuracy: 98.71% 27 Validation loss: 0.140362 Best loss: 0.052497 Accuracy: 98.83% 28 Validation loss: 0.212676 Best loss: 0.052497 Accuracy: 98.28% 29 Validation loss: 0.161250 Best loss: 0.052497 Accuracy: 98.67% 30 Validation loss: 0.153633 Best loss: 0.052497 Accuracy: 98.79% 31 Validation loss: 0.153754 Best loss: 0.052497 Accuracy: 98.75% 32 Validation loss: 0.141776 Best loss: 0.052497 Accuracy: 98.91% 33 Validation loss: 0.193727 Best loss: 0.052497 Accuracy: 98.83% 34 Validation loss: 0.178126 Best loss: 0.052497 Accuracy: 98.79% 35 Validation loss: 0.166841 Best loss: 0.052497 Accuracy: 98.71% 36 Validation loss: 0.162473 Best loss: 0.052497 Accuracy: 98.36% 37 Validation loss: 0.174592 Best loss: 0.052497 Accuracy: 98.63% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 47.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.191507 Best loss: 0.191507 Accuracy: 95.66% 1 Validation loss: 0.092628 Best loss: 0.092628 Accuracy: 97.42% 2 Validation loss: 0.088270 Best loss: 0.088270 Accuracy: 97.62% 3 Validation loss: 0.108749 Best loss: 0.088270 Accuracy: 97.50% 4 Validation loss: 0.081499 Best loss: 0.081499 Accuracy: 98.12% 5 Validation loss: 0.059363 Best loss: 0.059363 Accuracy: 98.51% 6 Validation loss: 0.077131 Best loss: 0.059363 Accuracy: 98.28% 7 Validation loss: 0.076968 Best loss: 0.059363 Accuracy: 98.16% 8 Validation loss: 0.060223 Best loss: 0.059363 Accuracy: 98.32% 9 Validation loss: 0.057619 Best loss: 0.057619 Accuracy: 98.55% 10 Validation loss: 0.085988 Best loss: 0.057619 Accuracy: 98.08% 11 Validation loss: 0.094348 Best loss: 0.057619 Accuracy: 98.08% 12 Validation loss: 0.059539 Best loss: 0.057619 Accuracy: 98.63% 13 Validation loss: 0.093883 Best loss: 0.057619 Accuracy: 97.58% 14 Validation loss: 0.065287 Best loss: 0.057619 Accuracy: 98.83% 15 Validation loss: 0.083307 Best loss: 0.057619 Accuracy: 98.48% 16 Validation loss: 0.059014 Best loss: 0.057619 Accuracy: 98.63% 17 Validation loss: 0.098305 Best loss: 0.057619 Accuracy: 98.55% 18 Validation loss: 0.153415 Best loss: 0.057619 Accuracy: 98.05% 19 Validation loss: 19.858881 Best loss: 0.057619 Accuracy: 86.08% 20 Validation loss: 0.125264 Best loss: 0.057619 Accuracy: 98.63% 21 Validation loss: 0.106003 Best loss: 0.057619 Accuracy: 98.59% 22 Validation loss: 0.089386 Best loss: 0.057619 Accuracy: 98.83% 23 Validation loss: 0.088383 Best loss: 0.057619 Accuracy: 98.79% 24 Validation loss: 0.069461 Best loss: 0.057619 Accuracy: 98.94% 25 Validation loss: 0.074238 Best loss: 0.057619 Accuracy: 98.87% 26 Validation loss: 0.064787 Best loss: 0.057619 Accuracy: 98.91% 27 Validation loss: 0.076463 Best loss: 0.057619 Accuracy: 99.06% 28 Validation loss: 0.136570 Best loss: 0.057619 Accuracy: 98.40% 29 Validation loss: 0.132457 Best loss: 0.057619 Accuracy: 98.28% 30 Validation loss: 0.054724 Best loss: 0.054724 Accuracy: 99.22% 31 Validation loss: 0.092548 Best loss: 0.054724 Accuracy: 98.87% 32 Validation loss: 0.062165 Best loss: 0.054724 Accuracy: 99.18% 33 Validation loss: 0.104894 Best loss: 0.054724 Accuracy: 98.67% 34 Validation loss: 0.083757 Best loss: 0.054724 Accuracy: 98.75% 35 Validation loss: 0.275576 Best loss: 0.054724 Accuracy: 98.32% 36 Validation loss: 0.913077 Best loss: 0.054724 Accuracy: 97.22% 37 Validation loss: 0.208319 Best loss: 0.054724 Accuracy: 98.71% 38 Validation loss: 0.120843 Best loss: 0.054724 Accuracy: 98.79% 39 Validation loss: 0.113708 Best loss: 0.054724 Accuracy: 98.71% 40 Validation loss: 0.107951 Best loss: 0.054724 Accuracy: 98.94% 41 Validation loss: 0.073375 Best loss: 0.054724 Accuracy: 98.98% 42 Validation loss: 0.108266 Best loss: 0.054724 Accuracy: 99.14% 43 Validation loss: 0.093624 Best loss: 0.054724 Accuracy: 99.02% 44 Validation loss: 0.094282 Best loss: 0.054724 Accuracy: 98.98% 45 Validation loss: 0.106172 Best loss: 0.054724 Accuracy: 99.10% 46 Validation loss: 0.086140 Best loss: 0.054724 Accuracy: 98.87% 47 Validation loss: 0.175984 Best loss: 0.054724 Accuracy: 98.83% 48 Validation loss: 0.144906 Best loss: 0.054724 Accuracy: 98.71% 49 Validation loss: 0.137024 Best loss: 0.054724 Accuracy: 98.91% 50 Validation loss: 0.132794 Best loss: 0.054724 Accuracy: 98.79% 51 Validation loss: 0.098684 Best loss: 0.054724 Accuracy: 98.71% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.1min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.255413 Best loss: 0.255413 Accuracy: 95.86% 1 Validation loss: 0.095087 Best loss: 0.095087 Accuracy: 97.50% 2 Validation loss: 0.096960 Best loss: 0.095087 Accuracy: 97.22% 3 Validation loss: 0.071437 Best loss: 0.071437 Accuracy: 98.20% 4 Validation loss: 0.070945 Best loss: 0.070945 Accuracy: 98.01% 5 Validation loss: 0.092578 Best loss: 0.070945 Accuracy: 97.69% 6 Validation loss: 0.073391 Best loss: 0.070945 Accuracy: 97.73% 7 Validation loss: 0.079488 Best loss: 0.070945 Accuracy: 98.24% 8 Validation loss: 0.058233 Best loss: 0.058233 Accuracy: 98.36% 9 Validation loss: 0.065057 Best loss: 0.058233 Accuracy: 98.40% 10 Validation loss: 0.058362 Best loss: 0.058233 Accuracy: 98.44% 11 Validation loss: 0.086593 Best loss: 0.058233 Accuracy: 98.28% 12 Validation loss: 0.066694 Best loss: 0.058233 Accuracy: 98.24% 13 Validation loss: 0.069168 Best loss: 0.058233 Accuracy: 98.51% 14 Validation loss: 0.053773 Best loss: 0.053773 Accuracy: 98.94% 15 Validation loss: 0.064306 Best loss: 0.053773 Accuracy: 98.75% 16 Validation loss: 0.083979 Best loss: 0.053773 Accuracy: 98.32% 17 Validation loss: 0.081734 Best loss: 0.053773 Accuracy: 98.24% 18 Validation loss: 0.065987 Best loss: 0.053773 Accuracy: 98.71% 19 Validation loss: 0.073137 Best loss: 0.053773 Accuracy: 98.55% 20 Validation loss: 0.078052 Best loss: 0.053773 Accuracy: 98.36% 21 Validation loss: 0.070196 Best loss: 0.053773 Accuracy: 98.48% 22 Validation loss: 0.095924 Best loss: 0.053773 Accuracy: 98.40% 23 Validation loss: 0.062888 Best loss: 0.053773 Accuracy: 98.83% 24 Validation loss: 0.053095 Best loss: 0.053095 Accuracy: 98.94% 25 Validation loss: 0.109526 Best loss: 0.053095 Accuracy: 97.97% 26 Validation loss: 0.080865 Best loss: 0.053095 Accuracy: 98.44% 27 Validation loss: 0.082276 Best loss: 0.053095 Accuracy: 98.20% 28 Validation loss: 0.071461 Best loss: 0.053095 Accuracy: 98.55% 29 Validation loss: 0.076913 Best loss: 0.053095 Accuracy: 98.48% 30 Validation loss: 0.056493 Best loss: 0.053095 Accuracy: 98.75% 31 Validation loss: 0.056952 Best loss: 0.053095 Accuracy: 98.71% 32 Validation loss: 0.058370 Best loss: 0.053095 Accuracy: 98.94% 33 Validation loss: 0.054573 Best loss: 0.053095 Accuracy: 99.02% 34 Validation loss: 0.055340 Best loss: 0.053095 Accuracy: 99.14% 35 Validation loss: 0.054599 Best loss: 0.053095 Accuracy: 99.10% 36 Validation loss: 0.055358 Best loss: 0.053095 Accuracy: 98.98% 37 Validation loss: 0.055205 Best loss: 0.053095 Accuracy: 99.02% 38 Validation loss: 0.053477 Best loss: 0.053095 Accuracy: 99.10% 39 Validation loss: 0.054261 Best loss: 0.053095 Accuracy: 99.10% 40 Validation loss: 0.055203 Best loss: 0.053095 Accuracy: 99.06% 41 Validation loss: 0.055728 Best loss: 0.053095 Accuracy: 99.06% 42 Validation loss: 0.055470 Best loss: 0.053095 Accuracy: 99.06% 43 Validation loss: 0.055644 Best loss: 0.053095 Accuracy: 99.06% 44 Validation loss: 0.055928 Best loss: 0.053095 Accuracy: 99.06% 45 Validation loss: 0.056384 Best loss: 0.053095 Accuracy: 99.06% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 14.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.292134 Best loss: 0.292134 Accuracy: 95.11% 1 Validation loss: 0.093392 Best loss: 0.093392 Accuracy: 97.73% 2 Validation loss: 0.074923 Best loss: 0.074923 Accuracy: 98.01% 3 Validation loss: 0.056392 Best loss: 0.056392 Accuracy: 98.36% 4 Validation loss: 0.090062 Best loss: 0.056392 Accuracy: 97.50% 5 Validation loss: 0.066480 Best loss: 0.056392 Accuracy: 97.81% 6 Validation loss: 0.066599 Best loss: 0.056392 Accuracy: 98.16% 7 Validation loss: 0.073543 Best loss: 0.056392 Accuracy: 98.12% 8 Validation loss: 0.070998 Best loss: 0.056392 Accuracy: 98.28% 9 Validation loss: 0.066846 Best loss: 0.056392 Accuracy: 98.48% 10 Validation loss: 0.081171 Best loss: 0.056392 Accuracy: 98.48% 11 Validation loss: 0.074845 Best loss: 0.056392 Accuracy: 98.32% 12 Validation loss: 0.069116 Best loss: 0.056392 Accuracy: 98.24% 13 Validation loss: 0.065804 Best loss: 0.056392 Accuracy: 98.40% 14 Validation loss: 0.056806 Best loss: 0.056392 Accuracy: 98.71% 15 Validation loss: 0.062326 Best loss: 0.056392 Accuracy: 98.32% 16 Validation loss: 0.057979 Best loss: 0.056392 Accuracy: 98.48% 17 Validation loss: 0.072600 Best loss: 0.056392 Accuracy: 98.48% 18 Validation loss: 0.073300 Best loss: 0.056392 Accuracy: 98.48% 19 Validation loss: 0.073644 Best loss: 0.056392 Accuracy: 98.40% 20 Validation loss: 0.068526 Best loss: 0.056392 Accuracy: 98.75% 21 Validation loss: 0.119198 Best loss: 0.056392 Accuracy: 97.69% 22 Validation loss: 0.076893 Best loss: 0.056392 Accuracy: 98.32% 23 Validation loss: 0.066132 Best loss: 0.056392 Accuracy: 98.71% 24 Validation loss: 0.059239 Best loss: 0.056392 Accuracy: 98.67% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 8.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.259217 Best loss: 0.259217 Accuracy: 95.97% 1 Validation loss: 0.082291 Best loss: 0.082291 Accuracy: 98.05% 2 Validation loss: 0.069846 Best loss: 0.069846 Accuracy: 97.81% 3 Validation loss: 0.067868 Best loss: 0.067868 Accuracy: 98.12% 4 Validation loss: 0.054622 Best loss: 0.054622 Accuracy: 98.59% 5 Validation loss: 0.082597 Best loss: 0.054622 Accuracy: 97.81% 6 Validation loss: 0.061903 Best loss: 0.054622 Accuracy: 98.40% 7 Validation loss: 0.053467 Best loss: 0.053467 Accuracy: 98.67% 8 Validation loss: 0.065131 Best loss: 0.053467 Accuracy: 98.32% 9 Validation loss: 0.062542 Best loss: 0.053467 Accuracy: 98.55% 10 Validation loss: 0.053353 Best loss: 0.053353 Accuracy: 98.59% 11 Validation loss: 0.072287 Best loss: 0.053353 Accuracy: 98.51% 12 Validation loss: 0.118744 Best loss: 0.053353 Accuracy: 97.69% 13 Validation loss: 0.121789 Best loss: 0.053353 Accuracy: 97.34% 14 Validation loss: 0.048696 Best loss: 0.048696 Accuracy: 98.87% 15 Validation loss: 0.050622 Best loss: 0.048696 Accuracy: 98.87% 16 Validation loss: 0.069716 Best loss: 0.048696 Accuracy: 98.63% 17 Validation loss: 0.075110 Best loss: 0.048696 Accuracy: 98.51% 18 Validation loss: 0.059577 Best loss: 0.048696 Accuracy: 98.83% 19 Validation loss: 0.053282 Best loss: 0.048696 Accuracy: 98.87% 20 Validation loss: 0.053553 Best loss: 0.048696 Accuracy: 98.91% 21 Validation loss: 0.061152 Best loss: 0.048696 Accuracy: 98.83% 22 Validation loss: 0.058196 Best loss: 0.048696 Accuracy: 98.94% 23 Validation loss: 0.050193 Best loss: 0.048696 Accuracy: 98.94% 24 Validation loss: 0.049599 Best loss: 0.048696 Accuracy: 99.14% 25 Validation loss: 0.095297 Best loss: 0.048696 Accuracy: 98.55% 26 Validation loss: 0.081316 Best loss: 0.048696 Accuracy: 98.44% 27 Validation loss: 0.093179 Best loss: 0.048696 Accuracy: 98.08% 28 Validation loss: 0.096469 Best loss: 0.048696 Accuracy: 98.28% 29 Validation loss: 0.077256 Best loss: 0.048696 Accuracy: 98.79% 30 Validation loss: 0.068180 Best loss: 0.048696 Accuracy: 98.87% 31 Validation loss: 0.057082 Best loss: 0.048696 Accuracy: 99.02% 32 Validation loss: 0.059512 Best loss: 0.048696 Accuracy: 98.91% 33 Validation loss: 0.049981 Best loss: 0.048696 Accuracy: 98.91% 34 Validation loss: 0.055286 Best loss: 0.048696 Accuracy: 98.87% 35 Validation loss: 0.060961 Best loss: 0.048696 Accuracy: 98.71% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 12.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.088194 Best loss: 0.088194 Accuracy: 97.22% 1 Validation loss: 0.070065 Best loss: 0.070065 Accuracy: 97.97% 2 Validation loss: 0.061615 Best loss: 0.061615 Accuracy: 98.12% 3 Validation loss: 0.046229 Best loss: 0.046229 Accuracy: 98.32% 4 Validation loss: 0.063796 Best loss: 0.046229 Accuracy: 98.12% 5 Validation loss: 0.059772 Best loss: 0.046229 Accuracy: 98.24% 6 Validation loss: 0.063068 Best loss: 0.046229 Accuracy: 97.93% 7 Validation loss: 0.054199 Best loss: 0.046229 Accuracy: 98.48% 8 Validation loss: 0.059249 Best loss: 0.046229 Accuracy: 98.59% 9 Validation loss: 0.051950 Best loss: 0.046229 Accuracy: 98.71% 10 Validation loss: 0.054640 Best loss: 0.046229 Accuracy: 98.67% 11 Validation loss: 0.031331 Best loss: 0.031331 Accuracy: 99.06% 12 Validation loss: 0.046665 Best loss: 0.031331 Accuracy: 98.67% 13 Validation loss: 0.047194 Best loss: 0.031331 Accuracy: 98.87% 14 Validation loss: 0.053651 Best loss: 0.031331 Accuracy: 98.59% 15 Validation loss: 0.054232 Best loss: 0.031331 Accuracy: 98.63% 16 Validation loss: 0.042859 Best loss: 0.031331 Accuracy: 98.71% 17 Validation loss: 0.047192 Best loss: 0.031331 Accuracy: 98.79% 18 Validation loss: 0.052321 Best loss: 0.031331 Accuracy: 98.75% 19 Validation loss: 0.066948 Best loss: 0.031331 Accuracy: 98.48% 20 Validation loss: 0.055696 Best loss: 0.031331 Accuracy: 98.63% 21 Validation loss: 0.055652 Best loss: 0.031331 Accuracy: 98.79% 22 Validation loss: 0.061219 Best loss: 0.031331 Accuracy: 98.59% 23 Validation loss: 0.065548 Best loss: 0.031331 Accuracy: 98.75% 24 Validation loss: 0.070788 Best loss: 0.031331 Accuracy: 98.79% 25 Validation loss: 0.047496 Best loss: 0.031331 Accuracy: 98.94% 26 Validation loss: 0.045052 Best loss: 0.031331 Accuracy: 98.79% 27 Validation loss: 0.055916 Best loss: 0.031331 Accuracy: 98.67% 28 Validation loss: 0.054249 Best loss: 0.031331 Accuracy: 98.44% 29 Validation loss: 0.086208 Best loss: 0.031331 Accuracy: 98.48% 30 Validation loss: 0.049357 Best loss: 0.031331 Accuracy: 98.94% 31 Validation loss: 0.057673 Best loss: 0.031331 Accuracy: 98.75% 32 Validation loss: 0.085559 Best loss: 0.031331 Accuracy: 98.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 47.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.077583 Best loss: 0.077583 Accuracy: 97.69% 1 Validation loss: 0.066486 Best loss: 0.066486 Accuracy: 97.73% 2 Validation loss: 0.052230 Best loss: 0.052230 Accuracy: 98.24% 3 Validation loss: 0.071256 Best loss: 0.052230 Accuracy: 98.32% 4 Validation loss: 0.068481 Best loss: 0.052230 Accuracy: 98.08% 5 Validation loss: 0.050005 Best loss: 0.050005 Accuracy: 98.59% 6 Validation loss: 0.056960 Best loss: 0.050005 Accuracy: 98.36% 7 Validation loss: 0.069099 Best loss: 0.050005 Accuracy: 97.97% 8 Validation loss: 0.058451 Best loss: 0.050005 Accuracy: 98.59% 9 Validation loss: 0.080398 Best loss: 0.050005 Accuracy: 98.16% 10 Validation loss: 0.078362 Best loss: 0.050005 Accuracy: 98.55% 11 Validation loss: 0.059265 Best loss: 0.050005 Accuracy: 98.48% 12 Validation loss: 0.059643 Best loss: 0.050005 Accuracy: 98.63% 13 Validation loss: 0.054417 Best loss: 0.050005 Accuracy: 98.63% 14 Validation loss: 0.040698 Best loss: 0.040698 Accuracy: 98.98% 15 Validation loss: 0.051754 Best loss: 0.040698 Accuracy: 98.75% 16 Validation loss: 0.056816 Best loss: 0.040698 Accuracy: 98.71% 17 Validation loss: 0.057797 Best loss: 0.040698 Accuracy: 98.59% 18 Validation loss: 0.059247 Best loss: 0.040698 Accuracy: 98.48% 19 Validation loss: 0.060535 Best loss: 0.040698 Accuracy: 98.67% 20 Validation loss: 0.063822 Best loss: 0.040698 Accuracy: 98.40% 21 Validation loss: 0.055186 Best loss: 0.040698 Accuracy: 98.71% 22 Validation loss: 0.054071 Best loss: 0.040698 Accuracy: 98.40% 23 Validation loss: 0.057487 Best loss: 0.040698 Accuracy: 98.51% 24 Validation loss: 0.042287 Best loss: 0.040698 Accuracy: 99.02% 25 Validation loss: 0.050257 Best loss: 0.040698 Accuracy: 98.83% 26 Validation loss: 0.049515 Best loss: 0.040698 Accuracy: 98.71% 27 Validation loss: 0.041967 Best loss: 0.040698 Accuracy: 99.10% 28 Validation loss: 0.091720 Best loss: 0.040698 Accuracy: 98.40% 29 Validation loss: 0.059404 Best loss: 0.040698 Accuracy: 98.71% 30 Validation loss: 0.049357 Best loss: 0.040698 Accuracy: 98.79% 31 Validation loss: 0.067991 Best loss: 0.040698 Accuracy: 98.67% 32 Validation loss: 0.059136 Best loss: 0.040698 Accuracy: 98.75% 33 Validation loss: 0.066086 Best loss: 0.040698 Accuracy: 98.59% 34 Validation loss: 0.047244 Best loss: 0.040698 Accuracy: 98.79% 35 Validation loss: 0.073341 Best loss: 0.040698 Accuracy: 98.28% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 52.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.090009 Best loss: 0.090009 Accuracy: 97.46% 1 Validation loss: 0.082743 Best loss: 0.082743 Accuracy: 97.62% 2 Validation loss: 0.081053 Best loss: 0.081053 Accuracy: 97.50% 3 Validation loss: 0.048408 Best loss: 0.048408 Accuracy: 98.71% 4 Validation loss: 0.052185 Best loss: 0.048408 Accuracy: 98.48% 5 Validation loss: 0.041557 Best loss: 0.041557 Accuracy: 98.71% 6 Validation loss: 0.055722 Best loss: 0.041557 Accuracy: 98.59% 7 Validation loss: 0.053313 Best loss: 0.041557 Accuracy: 98.63% 8 Validation loss: 0.052515 Best loss: 0.041557 Accuracy: 98.87% 9 Validation loss: 0.054673 Best loss: 0.041557 Accuracy: 98.40% 10 Validation loss: 0.056696 Best loss: 0.041557 Accuracy: 98.44% 11 Validation loss: 0.049218 Best loss: 0.041557 Accuracy: 98.67% 12 Validation loss: 0.041739 Best loss: 0.041557 Accuracy: 98.98% 13 Validation loss: 0.053821 Best loss: 0.041557 Accuracy: 98.79% 14 Validation loss: 0.051923 Best loss: 0.041557 Accuracy: 98.63% 15 Validation loss: 0.056510 Best loss: 0.041557 Accuracy: 98.67% 16 Validation loss: 0.050175 Best loss: 0.041557 Accuracy: 98.59% 17 Validation loss: 0.057657 Best loss: 0.041557 Accuracy: 98.71% 18 Validation loss: 0.050191 Best loss: 0.041557 Accuracy: 98.59% 19 Validation loss: 0.050944 Best loss: 0.041557 Accuracy: 98.79% 20 Validation loss: 0.052991 Best loss: 0.041557 Accuracy: 98.67% 21 Validation loss: 0.072864 Best loss: 0.041557 Accuracy: 98.75% 22 Validation loss: 0.051866 Best loss: 0.041557 Accuracy: 98.94% 23 Validation loss: 0.060846 Best loss: 0.041557 Accuracy: 98.55% 24 Validation loss: 0.040301 Best loss: 0.040301 Accuracy: 99.10% 25 Validation loss: 0.049084 Best loss: 0.040301 Accuracy: 98.94% 26 Validation loss: 0.044738 Best loss: 0.040301 Accuracy: 98.94% 27 Validation loss: 0.086362 Best loss: 0.040301 Accuracy: 98.16% 28 Validation loss: 0.059645 Best loss: 0.040301 Accuracy: 98.75% 29 Validation loss: 0.051391 Best loss: 0.040301 Accuracy: 98.87% 30 Validation loss: 0.044481 Best loss: 0.040301 Accuracy: 98.83% 31 Validation loss: 0.087608 Best loss: 0.040301 Accuracy: 98.55% 32 Validation loss: 0.061502 Best loss: 0.040301 Accuracy: 98.71% 33 Validation loss: 0.070123 Best loss: 0.040301 Accuracy: 98.67% 34 Validation loss: 0.087704 Best loss: 0.040301 Accuracy: 98.32% 35 Validation loss: 0.065733 Best loss: 0.040301 Accuracy: 98.75% 36 Validation loss: 0.045683 Best loss: 0.040301 Accuracy: 99.02% 37 Validation loss: 0.049229 Best loss: 0.040301 Accuracy: 98.79% 38 Validation loss: 0.060039 Best loss: 0.040301 Accuracy: 98.98% 39 Validation loss: 0.052207 Best loss: 0.040301 Accuracy: 99.02% 40 Validation loss: 0.058214 Best loss: 0.040301 Accuracy: 98.87% 41 Validation loss: 0.056860 Best loss: 0.040301 Accuracy: 98.59% 42 Validation loss: 0.055116 Best loss: 0.040301 Accuracy: 98.87% 43 Validation loss: 0.077575 Best loss: 0.040301 Accuracy: 98.79% 44 Validation loss: 0.067774 Best loss: 0.040301 Accuracy: 98.71% 45 Validation loss: 0.075019 Best loss: 0.040301 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 1.1min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.103969 Best loss: 0.103969 Accuracy: 96.64% 1 Validation loss: 0.087710 Best loss: 0.087710 Accuracy: 97.19% 2 Validation loss: 0.097128 Best loss: 0.087710 Accuracy: 96.79% 3 Validation loss: 0.071950 Best loss: 0.071950 Accuracy: 97.85% 4 Validation loss: 0.102288 Best loss: 0.071950 Accuracy: 96.99% 5 Validation loss: 0.071330 Best loss: 0.071330 Accuracy: 97.77% 6 Validation loss: 0.075233 Best loss: 0.071330 Accuracy: 97.62% 7 Validation loss: 0.078166 Best loss: 0.071330 Accuracy: 97.46% 8 Validation loss: 0.107001 Best loss: 0.071330 Accuracy: 97.07% 9 Validation loss: 0.073147 Best loss: 0.071330 Accuracy: 97.89% 10 Validation loss: 0.081494 Best loss: 0.071330 Accuracy: 97.69% 11 Validation loss: 0.067058 Best loss: 0.067058 Accuracy: 98.01% 12 Validation loss: 0.072170 Best loss: 0.067058 Accuracy: 97.89% 13 Validation loss: 0.080278 Best loss: 0.067058 Accuracy: 97.89% 14 Validation loss: 0.072165 Best loss: 0.067058 Accuracy: 97.81% 15 Validation loss: 0.078859 Best loss: 0.067058 Accuracy: 97.54% 16 Validation loss: 0.089484 Best loss: 0.067058 Accuracy: 97.81% 17 Validation loss: 0.072995 Best loss: 0.067058 Accuracy: 97.97% 18 Validation loss: 0.071615 Best loss: 0.067058 Accuracy: 98.05% 19 Validation loss: 0.075116 Best loss: 0.067058 Accuracy: 97.81% 20 Validation loss: 0.086819 Best loss: 0.067058 Accuracy: 97.85% 21 Validation loss: 0.070240 Best loss: 0.067058 Accuracy: 98.28% 22 Validation loss: 0.073812 Best loss: 0.067058 Accuracy: 98.24% 23 Validation loss: 0.075448 Best loss: 0.067058 Accuracy: 97.97% 24 Validation loss: 0.081558 Best loss: 0.067058 Accuracy: 98.01% 25 Validation loss: 0.086653 Best loss: 0.067058 Accuracy: 97.97% 26 Validation loss: 0.082764 Best loss: 0.067058 Accuracy: 98.01% 27 Validation loss: 0.080194 Best loss: 0.067058 Accuracy: 98.05% 28 Validation loss: 0.092612 Best loss: 0.067058 Accuracy: 97.81% 29 Validation loss: 0.085792 Best loss: 0.067058 Accuracy: 97.89% 30 Validation loss: 0.089059 Best loss: 0.067058 Accuracy: 97.46% 31 Validation loss: 0.109665 Best loss: 0.067058 Accuracy: 97.38% 32 Validation loss: 0.086730 Best loss: 0.067058 Accuracy: 98.16% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 42.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.142314 Best loss: 0.142314 Accuracy: 95.43% 1 Validation loss: 0.085669 Best loss: 0.085669 Accuracy: 97.30% 2 Validation loss: 0.073518 Best loss: 0.073518 Accuracy: 97.54% 3 Validation loss: 0.102196 Best loss: 0.073518 Accuracy: 96.68% 4 Validation loss: 0.070195 Best loss: 0.070195 Accuracy: 97.89% 5 Validation loss: 0.079325 Best loss: 0.070195 Accuracy: 97.62% 6 Validation loss: 0.083252 Best loss: 0.070195 Accuracy: 97.85% 7 Validation loss: 0.077744 Best loss: 0.070195 Accuracy: 97.69% 8 Validation loss: 0.064170 Best loss: 0.064170 Accuracy: 98.12% 9 Validation loss: 0.165426 Best loss: 0.064170 Accuracy: 94.68% 10 Validation loss: 0.073553 Best loss: 0.064170 Accuracy: 98.05% 11 Validation loss: 0.075892 Best loss: 0.064170 Accuracy: 98.01% 12 Validation loss: 0.071764 Best loss: 0.064170 Accuracy: 98.01% 13 Validation loss: 0.073754 Best loss: 0.064170 Accuracy: 98.01% 14 Validation loss: 0.061390 Best loss: 0.061390 Accuracy: 98.32% 15 Validation loss: 0.081141 Best loss: 0.061390 Accuracy: 98.05% 16 Validation loss: 0.086735 Best loss: 0.061390 Accuracy: 97.73% 17 Validation loss: 0.080096 Best loss: 0.061390 Accuracy: 97.85% 18 Validation loss: 0.074262 Best loss: 0.061390 Accuracy: 98.08% 19 Validation loss: 0.075258 Best loss: 0.061390 Accuracy: 97.89% 20 Validation loss: 0.080237 Best loss: 0.061390 Accuracy: 98.01% 21 Validation loss: 0.082724 Best loss: 0.061390 Accuracy: 97.81% 22 Validation loss: 0.128365 Best loss: 0.061390 Accuracy: 96.25% 23 Validation loss: 0.087615 Best loss: 0.061390 Accuracy: 97.93% 24 Validation loss: 0.062475 Best loss: 0.061390 Accuracy: 98.44% 25 Validation loss: 0.085750 Best loss: 0.061390 Accuracy: 97.93% 26 Validation loss: 0.104315 Best loss: 0.061390 Accuracy: 97.26% 27 Validation loss: 0.063547 Best loss: 0.061390 Accuracy: 98.20% 28 Validation loss: 0.093927 Best loss: 0.061390 Accuracy: 97.65% 29 Validation loss: 0.081855 Best loss: 0.061390 Accuracy: 98.01% 30 Validation loss: 0.082730 Best loss: 0.061390 Accuracy: 98.05% 31 Validation loss: 0.063871 Best loss: 0.061390 Accuracy: 98.32% 32 Validation loss: 0.082373 Best loss: 0.061390 Accuracy: 98.20% 33 Validation loss: 0.090219 Best loss: 0.061390 Accuracy: 97.93% 34 Validation loss: 0.081150 Best loss: 0.061390 Accuracy: 98.36% 35 Validation loss: 0.076000 Best loss: 0.061390 Accuracy: 97.97% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 43.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.105640 Best loss: 0.105640 Accuracy: 97.11% 1 Validation loss: 0.118222 Best loss: 0.105640 Accuracy: 95.93% 2 Validation loss: 0.078576 Best loss: 0.078576 Accuracy: 97.34% 3 Validation loss: 0.062884 Best loss: 0.062884 Accuracy: 98.16% 4 Validation loss: 0.081941 Best loss: 0.062884 Accuracy: 97.73% 5 Validation loss: 0.091959 Best loss: 0.062884 Accuracy: 97.34% 6 Validation loss: 0.085539 Best loss: 0.062884 Accuracy: 97.54% 7 Validation loss: 0.078001 Best loss: 0.062884 Accuracy: 97.58% 8 Validation loss: 0.129663 Best loss: 0.062884 Accuracy: 96.83% 9 Validation loss: 0.075293 Best loss: 0.062884 Accuracy: 97.85% 10 Validation loss: 0.074905 Best loss: 0.062884 Accuracy: 97.77% 11 Validation loss: 0.059306 Best loss: 0.059306 Accuracy: 98.05% 12 Validation loss: 0.065483 Best loss: 0.059306 Accuracy: 98.01% 13 Validation loss: 0.064614 Best loss: 0.059306 Accuracy: 98.36% 14 Validation loss: 0.070155 Best loss: 0.059306 Accuracy: 98.20% 15 Validation loss: 0.082729 Best loss: 0.059306 Accuracy: 97.77% 16 Validation loss: 0.063477 Best loss: 0.059306 Accuracy: 98.32% 17 Validation loss: 0.065452 Best loss: 0.059306 Accuracy: 98.20% 18 Validation loss: 0.085616 Best loss: 0.059306 Accuracy: 97.81% 19 Validation loss: 0.085740 Best loss: 0.059306 Accuracy: 97.77% 20 Validation loss: 0.072945 Best loss: 0.059306 Accuracy: 98.08% 21 Validation loss: 0.069948 Best loss: 0.059306 Accuracy: 98.12% 22 Validation loss: 0.074157 Best loss: 0.059306 Accuracy: 98.24% 23 Validation loss: 0.072424 Best loss: 0.059306 Accuracy: 98.36% 24 Validation loss: 0.073368 Best loss: 0.059306 Accuracy: 98.36% 25 Validation loss: 0.073043 Best loss: 0.059306 Accuracy: 98.36% 26 Validation loss: 0.070829 Best loss: 0.059306 Accuracy: 98.24% 27 Validation loss: 0.080964 Best loss: 0.059306 Accuracy: 97.97% 28 Validation loss: 0.075490 Best loss: 0.059306 Accuracy: 98.40% 29 Validation loss: 0.078049 Best loss: 0.059306 Accuracy: 97.97% 30 Validation loss: 0.078529 Best loss: 0.059306 Accuracy: 98.16% 31 Validation loss: 0.079089 Best loss: 0.059306 Accuracy: 98.12% 32 Validation loss: 0.094173 Best loss: 0.059306 Accuracy: 97.93% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 40.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.117448 Best loss: 0.117448 Accuracy: 96.60% 1 Validation loss: 0.076707 Best loss: 0.076707 Accuracy: 97.69% 2 Validation loss: 0.094360 Best loss: 0.076707 Accuracy: 96.83% 3 Validation loss: 0.075655 Best loss: 0.075655 Accuracy: 97.62% 4 Validation loss: 0.063814 Best loss: 0.063814 Accuracy: 98.05% 5 Validation loss: 0.078804 Best loss: 0.063814 Accuracy: 97.81% 6 Validation loss: 0.070745 Best loss: 0.063814 Accuracy: 97.69% 7 Validation loss: 0.072708 Best loss: 0.063814 Accuracy: 97.42% 8 Validation loss: 0.104306 Best loss: 0.063814 Accuracy: 97.07% 9 Validation loss: 0.057308 Best loss: 0.057308 Accuracy: 98.05% 10 Validation loss: 0.065713 Best loss: 0.057308 Accuracy: 98.28% 11 Validation loss: 0.061215 Best loss: 0.057308 Accuracy: 98.32% 12 Validation loss: 0.061483 Best loss: 0.057308 Accuracy: 98.12% 13 Validation loss: 0.088551 Best loss: 0.057308 Accuracy: 97.58% 14 Validation loss: 0.061269 Best loss: 0.057308 Accuracy: 98.36% 15 Validation loss: 0.066395 Best loss: 0.057308 Accuracy: 98.28% 16 Validation loss: 0.065537 Best loss: 0.057308 Accuracy: 98.08% 17 Validation loss: 0.074780 Best loss: 0.057308 Accuracy: 98.20% 18 Validation loss: 0.070059 Best loss: 0.057308 Accuracy: 98.05% 19 Validation loss: 0.062966 Best loss: 0.057308 Accuracy: 98.48% 20 Validation loss: 0.074110 Best loss: 0.057308 Accuracy: 98.12% 21 Validation loss: 0.057675 Best loss: 0.057308 Accuracy: 98.40% 22 Validation loss: 0.053459 Best loss: 0.053459 Accuracy: 98.48% 23 Validation loss: 0.059237 Best loss: 0.053459 Accuracy: 98.51% 24 Validation loss: 0.062955 Best loss: 0.053459 Accuracy: 98.28% 25 Validation loss: 0.070734 Best loss: 0.053459 Accuracy: 98.12% 26 Validation loss: 0.057572 Best loss: 0.053459 Accuracy: 98.36% 27 Validation loss: 0.068742 Best loss: 0.053459 Accuracy: 98.20% 28 Validation loss: 0.065845 Best loss: 0.053459 Accuracy: 98.40% 29 Validation loss: 0.069608 Best loss: 0.053459 Accuracy: 98.40% 30 Validation loss: 0.065007 Best loss: 0.053459 Accuracy: 98.44% 31 Validation loss: 0.078970 Best loss: 0.053459 Accuracy: 98.05% 32 Validation loss: 0.084031 Best loss: 0.053459 Accuracy: 97.93% 33 Validation loss: 0.065570 Best loss: 0.053459 Accuracy: 98.32% 34 Validation loss: 0.086542 Best loss: 0.053459 Accuracy: 98.08% 35 Validation loss: 0.056244 Best loss: 0.053459 Accuracy: 98.51% 36 Validation loss: 0.078165 Best loss: 0.053459 Accuracy: 98.48% 37 Validation loss: 0.077689 Best loss: 0.053459 Accuracy: 98.24% 38 Validation loss: 0.083363 Best loss: 0.053459 Accuracy: 98.32% 39 Validation loss: 0.075358 Best loss: 0.053459 Accuracy: 98.12% 40 Validation loss: 0.081186 Best loss: 0.053459 Accuracy: 98.12% 41 Validation loss: 0.090372 Best loss: 0.053459 Accuracy: 97.73% 42 Validation loss: 0.077881 Best loss: 0.053459 Accuracy: 98.24% 43 Validation loss: 0.071159 Best loss: 0.053459 Accuracy: 98.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 1.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.187258 Best loss: 0.187258 Accuracy: 94.68% 1 Validation loss: 0.082004 Best loss: 0.082004 Accuracy: 97.73% 2 Validation loss: 0.068555 Best loss: 0.068555 Accuracy: 98.08% 3 Validation loss: 0.127596 Best loss: 0.068555 Accuracy: 95.66% 4 Validation loss: 0.059655 Best loss: 0.059655 Accuracy: 98.12% 5 Validation loss: 0.053595 Best loss: 0.053595 Accuracy: 98.20% 6 Validation loss: 0.072172 Best loss: 0.053595 Accuracy: 97.97% 7 Validation loss: 0.067103 Best loss: 0.053595 Accuracy: 98.16% 8 Validation loss: 0.060136 Best loss: 0.053595 Accuracy: 98.40% 9 Validation loss: 0.065278 Best loss: 0.053595 Accuracy: 98.16% 10 Validation loss: 0.071561 Best loss: 0.053595 Accuracy: 97.77% 11 Validation loss: 0.069094 Best loss: 0.053595 Accuracy: 98.16% 12 Validation loss: 0.058282 Best loss: 0.053595 Accuracy: 98.40% 13 Validation loss: 0.074202 Best loss: 0.053595 Accuracy: 98.01% 14 Validation loss: 0.053586 Best loss: 0.053586 Accuracy: 98.63% 15 Validation loss: 0.056644 Best loss: 0.053586 Accuracy: 98.40% 16 Validation loss: 0.097638 Best loss: 0.053586 Accuracy: 97.54% 17 Validation loss: 0.061058 Best loss: 0.053586 Accuracy: 98.16% 18 Validation loss: 0.072351 Best loss: 0.053586 Accuracy: 98.01% 19 Validation loss: 0.060882 Best loss: 0.053586 Accuracy: 98.24% 20 Validation loss: 0.054885 Best loss: 0.053586 Accuracy: 98.36% 21 Validation loss: 0.064842 Best loss: 0.053586 Accuracy: 98.20% 22 Validation loss: 0.070709 Best loss: 0.053586 Accuracy: 98.20% 23 Validation loss: 0.062034 Best loss: 0.053586 Accuracy: 98.40% 24 Validation loss: 0.055747 Best loss: 0.053586 Accuracy: 98.44% 25 Validation loss: 0.087580 Best loss: 0.053586 Accuracy: 97.97% 26 Validation loss: 0.073201 Best loss: 0.053586 Accuracy: 98.48% 27 Validation loss: 0.068062 Best loss: 0.053586 Accuracy: 98.28% 28 Validation loss: 0.065222 Best loss: 0.053586 Accuracy: 98.32% 29 Validation loss: 0.089082 Best loss: 0.053586 Accuracy: 98.05% 30 Validation loss: 0.078439 Best loss: 0.053586 Accuracy: 98.20% 31 Validation loss: 0.083307 Best loss: 0.053586 Accuracy: 98.05% 32 Validation loss: 0.056977 Best loss: 0.053586 Accuracy: 98.44% 33 Validation loss: 0.075990 Best loss: 0.053586 Accuracy: 98.12% 34 Validation loss: 0.068682 Best loss: 0.053586 Accuracy: 98.28% 35 Validation loss: 0.092126 Best loss: 0.053586 Accuracy: 97.89% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 46.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.093096 Best loss: 0.093096 Accuracy: 97.42% 1 Validation loss: 0.076971 Best loss: 0.076971 Accuracy: 97.81% 2 Validation loss: 0.066792 Best loss: 0.066792 Accuracy: 97.89% 3 Validation loss: 0.057694 Best loss: 0.057694 Accuracy: 98.32% 4 Validation loss: 0.065398 Best loss: 0.057694 Accuracy: 98.01% 5 Validation loss: 0.062746 Best loss: 0.057694 Accuracy: 98.20% 6 Validation loss: 0.054335 Best loss: 0.054335 Accuracy: 98.51% 7 Validation loss: 0.074550 Best loss: 0.054335 Accuracy: 97.54% 8 Validation loss: 0.075018 Best loss: 0.054335 Accuracy: 98.05% 9 Validation loss: 0.059359 Best loss: 0.054335 Accuracy: 98.24% 10 Validation loss: 0.052549 Best loss: 0.052549 Accuracy: 98.59% 11 Validation loss: 0.062185 Best loss: 0.052549 Accuracy: 98.44% 12 Validation loss: 0.071690 Best loss: 0.052549 Accuracy: 98.44% 13 Validation loss: 0.061438 Best loss: 0.052549 Accuracy: 98.12% 14 Validation loss: 0.074290 Best loss: 0.052549 Accuracy: 97.97% 15 Validation loss: 0.089562 Best loss: 0.052549 Accuracy: 97.97% 16 Validation loss: 0.071082 Best loss: 0.052549 Accuracy: 98.32% 17 Validation loss: 0.074349 Best loss: 0.052549 Accuracy: 97.89% 18 Validation loss: 0.067977 Best loss: 0.052549 Accuracy: 98.44% 19 Validation loss: 0.074460 Best loss: 0.052549 Accuracy: 98.44% 20 Validation loss: 0.067762 Best loss: 0.052549 Accuracy: 98.67% 21 Validation loss: 0.057713 Best loss: 0.052549 Accuracy: 98.55% 22 Validation loss: 0.078986 Best loss: 0.052549 Accuracy: 98.32% 23 Validation loss: 0.067824 Best loss: 0.052549 Accuracy: 98.59% 24 Validation loss: 0.065871 Best loss: 0.052549 Accuracy: 98.44% 25 Validation loss: 0.077946 Best loss: 0.052549 Accuracy: 98.12% 26 Validation loss: 0.065905 Best loss: 0.052549 Accuracy: 98.79% 27 Validation loss: 0.063598 Best loss: 0.052549 Accuracy: 98.36% 28 Validation loss: 0.068421 Best loss: 0.052549 Accuracy: 98.40% 29 Validation loss: 0.070194 Best loss: 0.052549 Accuracy: 98.36% 30 Validation loss: 0.061322 Best loss: 0.052549 Accuracy: 98.59% 31 Validation loss: 0.082005 Best loss: 0.052549 Accuracy: 98.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=10, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 44.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.478592 Best loss: 0.478592 Accuracy: 97.07% 1 Validation loss: 0.347604 Best loss: 0.347604 Accuracy: 96.36% 2 Validation loss: 0.518059 Best loss: 0.347604 Accuracy: 96.01% 3 Validation loss: 0.174067 Best loss: 0.174067 Accuracy: 97.89% 4 Validation loss: 0.141108 Best loss: 0.141108 Accuracy: 98.08% 5 Validation loss: 0.158516 Best loss: 0.141108 Accuracy: 98.44% 6 Validation loss: 0.108217 Best loss: 0.108217 Accuracy: 98.16% 7 Validation loss: 0.081850 Best loss: 0.081850 Accuracy: 99.06% 8 Validation loss: 0.088707 Best loss: 0.081850 Accuracy: 98.59% 9 Validation loss: 0.082421 Best loss: 0.081850 Accuracy: 98.48% 10 Validation loss: 0.081759 Best loss: 0.081759 Accuracy: 98.98% 11 Validation loss: 0.061672 Best loss: 0.061672 Accuracy: 99.06% 12 Validation loss: 0.090300 Best loss: 0.061672 Accuracy: 98.79% 13 Validation loss: 0.115268 Best loss: 0.061672 Accuracy: 98.63% 14 Validation loss: 0.071469 Best loss: 0.061672 Accuracy: 98.71% 15 Validation loss: 0.052572 Best loss: 0.052572 Accuracy: 99.14% 16 Validation loss: 0.060699 Best loss: 0.052572 Accuracy: 98.75% 17 Validation loss: 0.091967 Best loss: 0.052572 Accuracy: 98.24% 18 Validation loss: 0.078099 Best loss: 0.052572 Accuracy: 98.94% 19 Validation loss: 0.056849 Best loss: 0.052572 Accuracy: 98.75% 20 Validation loss: 0.077825 Best loss: 0.052572 Accuracy: 98.83% 21 Validation loss: 0.111760 Best loss: 0.052572 Accuracy: 98.40% 22 Validation loss: 0.065810 Best loss: 0.052572 Accuracy: 98.91% 23 Validation loss: 0.093976 Best loss: 0.052572 Accuracy: 98.40% 24 Validation loss: 0.067365 Best loss: 0.052572 Accuracy: 98.87% 25 Validation loss: 0.059211 Best loss: 0.052572 Accuracy: 98.71% 26 Validation loss: 0.070340 Best loss: 0.052572 Accuracy: 98.75% 27 Validation loss: 0.050476 Best loss: 0.050476 Accuracy: 98.98% 28 Validation loss: 0.089140 Best loss: 0.050476 Accuracy: 98.67% 29 Validation loss: 0.067998 Best loss: 0.050476 Accuracy: 98.91% 30 Validation loss: 0.067486 Best loss: 0.050476 Accuracy: 98.91% 31 Validation loss: 0.079156 Best loss: 0.050476 Accuracy: 98.55% 32 Validation loss: 0.062881 Best loss: 0.050476 Accuracy: 98.55% 33 Validation loss: 0.080346 Best loss: 0.050476 Accuracy: 98.63% 34 Validation loss: 0.065497 Best loss: 0.050476 Accuracy: 98.79% 35 Validation loss: 0.058225 Best loss: 0.050476 Accuracy: 98.87% 36 Validation loss: 0.055799 Best loss: 0.050476 Accuracy: 98.83% 37 Validation loss: 0.095220 Best loss: 0.050476 Accuracy: 98.08% 38 Validation loss: 0.055916 Best loss: 0.050476 Accuracy: 99.10% 39 Validation loss: 0.096943 Best loss: 0.050476 Accuracy: 98.48% 40 Validation loss: 0.073656 Best loss: 0.050476 Accuracy: 98.79% 41 Validation loss: 0.088299 Best loss: 0.050476 Accuracy: 98.79% 42 Validation loss: 0.070914 Best loss: 0.050476 Accuracy: 98.87% 43 Validation loss: 0.074631 Best loss: 0.050476 Accuracy: 98.98% 44 Validation loss: 0.065926 Best loss: 0.050476 Accuracy: 99.14% 45 Validation loss: 0.085774 Best loss: 0.050476 Accuracy: 98.67% 46 Validation loss: 0.061712 Best loss: 0.050476 Accuracy: 98.98% 47 Validation loss: 0.063011 Best loss: 0.050476 Accuracy: 98.75% 48 Validation loss: 0.080130 Best loss: 0.050476 Accuracy: 98.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01, total= 2.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.666906 Best loss: 0.666906 Accuracy: 95.50% 1 Validation loss: 0.273613 Best loss: 0.273613 Accuracy: 98.08% 2 Validation loss: 0.255826 Best loss: 0.255826 Accuracy: 97.58% 3 Validation loss: 0.254037 Best loss: 0.254037 Accuracy: 97.38% 4 Validation loss: 0.146267 Best loss: 0.146267 Accuracy: 98.24% 5 Validation loss: 0.151236 Best loss: 0.146267 Accuracy: 98.24% 6 Validation loss: 0.098711 Best loss: 0.098711 Accuracy: 98.63% 7 Validation loss: 0.112843 Best loss: 0.098711 Accuracy: 98.24% 8 Validation loss: 0.093207 Best loss: 0.093207 Accuracy: 98.83% 9 Validation loss: 0.118634 Best loss: 0.093207 Accuracy: 98.44% 10 Validation loss: 0.137945 Best loss: 0.093207 Accuracy: 97.89% 11 Validation loss: 0.083620 Best loss: 0.083620 Accuracy: 98.71% 12 Validation loss: 0.100558 Best loss: 0.083620 Accuracy: 98.32% 13 Validation loss: 0.093642 Best loss: 0.083620 Accuracy: 98.71% 14 Validation loss: 0.099482 Best loss: 0.083620 Accuracy: 98.40% 15 Validation loss: 0.118163 Best loss: 0.083620 Accuracy: 98.32% 16 Validation loss: 0.101624 Best loss: 0.083620 Accuracy: 98.36% 17 Validation loss: 0.078886 Best loss: 0.078886 Accuracy: 98.79% 18 Validation loss: 0.081456 Best loss: 0.078886 Accuracy: 98.67% 19 Validation loss: 0.126715 Best loss: 0.078886 Accuracy: 98.32% 20 Validation loss: 0.072169 Best loss: 0.072169 Accuracy: 98.67% 21 Validation loss: 0.079924 Best loss: 0.072169 Accuracy: 98.94% 22 Validation loss: 0.065758 Best loss: 0.065758 Accuracy: 98.91% 23 Validation loss: 0.099291 Best loss: 0.065758 Accuracy: 98.40% 24 Validation loss: 0.089774 Best loss: 0.065758 Accuracy: 98.67% 25 Validation loss: 0.059461 Best loss: 0.059461 Accuracy: 98.94% 26 Validation loss: 0.093074 Best loss: 0.059461 Accuracy: 98.48% 27 Validation loss: 0.091897 Best loss: 0.059461 Accuracy: 98.75% 28 Validation loss: 0.085298 Best loss: 0.059461 Accuracy: 98.79% 29 Validation loss: 0.073832 Best loss: 0.059461 Accuracy: 98.83% 30 Validation loss: 0.091625 Best loss: 0.059461 Accuracy: 98.63% 31 Validation loss: 0.089453 Best loss: 0.059461 Accuracy: 98.75% 32 Validation loss: 0.058109 Best loss: 0.058109 Accuracy: 98.71% 33 Validation loss: 0.073891 Best loss: 0.058109 Accuracy: 98.59% 34 Validation loss: 0.068922 Best loss: 0.058109 Accuracy: 98.83% 35 Validation loss: 0.072536 Best loss: 0.058109 Accuracy: 98.94% 36 Validation loss: 0.083519 Best loss: 0.058109 Accuracy: 98.67% 37 Validation loss: 0.072777 Best loss: 0.058109 Accuracy: 98.91% 38 Validation loss: 0.065745 Best loss: 0.058109 Accuracy: 98.94% 39 Validation loss: 0.088205 Best loss: 0.058109 Accuracy: 98.75% 40 Validation loss: 0.069249 Best loss: 0.058109 Accuracy: 98.79% 41 Validation loss: 0.057447 Best loss: 0.057447 Accuracy: 98.98% 42 Validation loss: 0.108597 Best loss: 0.057447 Accuracy: 98.55% 43 Validation loss: 0.084982 Best loss: 0.057447 Accuracy: 98.94% 44 Validation loss: 0.130524 Best loss: 0.057447 Accuracy: 98.32% 45 Validation loss: 0.068925 Best loss: 0.057447 Accuracy: 99.14% 46 Validation loss: 0.074110 Best loss: 0.057447 Accuracy: 98.98% 47 Validation loss: 0.078699 Best loss: 0.057447 Accuracy: 98.67% 48 Validation loss: 0.087135 Best loss: 0.057447 Accuracy: 98.94% 49 Validation loss: 0.063436 Best loss: 0.057447 Accuracy: 98.91% 50 Validation loss: 0.080761 Best loss: 0.057447 Accuracy: 98.79% 51 Validation loss: 0.062664 Best loss: 0.057447 Accuracy: 99.18% 52 Validation loss: 0.080815 Best loss: 0.057447 Accuracy: 98.67% 53 Validation loss: 0.113576 Best loss: 0.057447 Accuracy: 98.44% 54 Validation loss: 0.077683 Best loss: 0.057447 Accuracy: 98.94% 55 Validation loss: 0.064104 Best loss: 0.057447 Accuracy: 99.02% 56 Validation loss: 0.084812 Best loss: 0.057447 Accuracy: 98.71% 57 Validation loss: 0.052847 Best loss: 0.052847 Accuracy: 99.02% 58 Validation loss: 0.066073 Best loss: 0.052847 Accuracy: 98.79% 59 Validation loss: 0.079854 Best loss: 0.052847 Accuracy: 98.94% 60 Validation loss: 0.073981 Best loss: 0.052847 Accuracy: 98.98% 61 Validation loss: 0.064028 Best loss: 0.052847 Accuracy: 99.02% 62 Validation loss: 0.055120 Best loss: 0.052847 Accuracy: 98.94% 63 Validation loss: 0.052429 Best loss: 0.052429 Accuracy: 99.06% 64 Validation loss: 0.054817 Best loss: 0.052429 Accuracy: 99.06% 65 Validation loss: 0.056670 Best loss: 0.052429 Accuracy: 98.94% 66 Validation loss: 0.057849 Best loss: 0.052429 Accuracy: 99.10% 67 Validation loss: 0.049187 Best loss: 0.049187 Accuracy: 99.14% 68 Validation loss: 0.061812 Best loss: 0.049187 Accuracy: 98.98% 69 Validation loss: 0.084929 Best loss: 0.049187 Accuracy: 98.91% 70 Validation loss: 0.075255 Best loss: 0.049187 Accuracy: 98.71% 71 Validation loss: 0.068245 Best loss: 0.049187 Accuracy: 98.91% 72 Validation loss: 0.060358 Best loss: 0.049187 Accuracy: 98.87% 73 Validation loss: 0.051804 Best loss: 0.049187 Accuracy: 98.94% 74 Validation loss: 0.065008 Best loss: 0.049187 Accuracy: 98.91% 75 Validation loss: 0.045871 Best loss: 0.045871 Accuracy: 99.10% 76 Validation loss: 0.048934 Best loss: 0.045871 Accuracy: 99.06% 77 Validation loss: 0.087652 Best loss: 0.045871 Accuracy: 98.71% 78 Validation loss: 0.078364 Best loss: 0.045871 Accuracy: 98.87% 79 Validation loss: 0.061607 Best loss: 0.045871 Accuracy: 98.87% 80 Validation loss: 0.084665 Best loss: 0.045871 Accuracy: 98.98% 81 Validation loss: 0.060284 Best loss: 0.045871 Accuracy: 99.02% 82 Validation loss: 0.063413 Best loss: 0.045871 Accuracy: 99.02% 83 Validation loss: 0.130204 Best loss: 0.045871 Accuracy: 98.51% 84 Validation loss: 0.074921 Best loss: 0.045871 Accuracy: 98.87% 85 Validation loss: 0.070944 Best loss: 0.045871 Accuracy: 98.87% 86 Validation loss: 0.075307 Best loss: 0.045871 Accuracy: 98.83% 87 Validation loss: 0.088891 Best loss: 0.045871 Accuracy: 98.48% 88 Validation loss: 0.077888 Best loss: 0.045871 Accuracy: 98.71% 89 Validation loss: 0.069346 Best loss: 0.045871 Accuracy: 98.59% 90 Validation loss: 0.067979 Best loss: 0.045871 Accuracy: 98.79% 91 Validation loss: 0.070868 Best loss: 0.045871 Accuracy: 98.71% 92 Validation loss: 0.073791 Best loss: 0.045871 Accuracy: 98.75% 93 Validation loss: 0.042376 Best loss: 0.042376 Accuracy: 99.02% 94 Validation loss: 0.054354 Best loss: 0.042376 Accuracy: 98.83% 95 Validation loss: 0.057862 Best loss: 0.042376 Accuracy: 98.79% 96 Validation loss: 0.052632 Best loss: 0.042376 Accuracy: 98.98% 97 Validation loss: 0.060970 Best loss: 0.042376 Accuracy: 99.06% 98 Validation loss: 0.058015 Best loss: 0.042376 Accuracy: 98.91% 99 Validation loss: 0.059868 Best loss: 0.042376 Accuracy: 98.94% 100 Validation loss: 0.060965 Best loss: 0.042376 Accuracy: 98.87% 101 Validation loss: 0.087156 Best loss: 0.042376 Accuracy: 98.75% 102 Validation loss: 0.063130 Best loss: 0.042376 Accuracy: 98.83% 103 Validation loss: 0.068124 Best loss: 0.042376 Accuracy: 99.06% 104 Validation loss: 0.105535 Best loss: 0.042376 Accuracy: 98.59% 105 Validation loss: 0.062022 Best loss: 0.042376 Accuracy: 98.87% 106 Validation loss: 0.085355 Best loss: 0.042376 Accuracy: 98.71% 107 Validation loss: 0.079946 Best loss: 0.042376 Accuracy: 98.79% 108 Validation loss: 0.054323 Best loss: 0.042376 Accuracy: 98.98% 109 Validation loss: 0.059667 Best loss: 0.042376 Accuracy: 99.02% 110 Validation loss: 0.064196 Best loss: 0.042376 Accuracy: 98.91% 111 Validation loss: 0.068348 Best loss: 0.042376 Accuracy: 98.75% 112 Validation loss: 0.047910 Best loss: 0.042376 Accuracy: 99.02% 113 Validation loss: 0.071105 Best loss: 0.042376 Accuracy: 98.87% 114 Validation loss: 0.083886 Best loss: 0.042376 Accuracy: 98.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01, total= 4.8min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.379027 Best loss: 0.379027 Accuracy: 96.33% 1 Validation loss: 0.568148 Best loss: 0.379027 Accuracy: 95.93% 2 Validation loss: 0.389170 Best loss: 0.379027 Accuracy: 96.44% 3 Validation loss: 0.193754 Best loss: 0.193754 Accuracy: 98.01% 4 Validation loss: 0.141584 Best loss: 0.141584 Accuracy: 98.36% 5 Validation loss: 0.149840 Best loss: 0.141584 Accuracy: 98.16% 6 Validation loss: 0.075422 Best loss: 0.075422 Accuracy: 98.71% 7 Validation loss: 0.168760 Best loss: 0.075422 Accuracy: 98.20% 8 Validation loss: 0.081242 Best loss: 0.075422 Accuracy: 98.59% 9 Validation loss: 0.052236 Best loss: 0.052236 Accuracy: 98.94% 10 Validation loss: 0.191423 Best loss: 0.052236 Accuracy: 98.12% 11 Validation loss: 0.093716 Best loss: 0.052236 Accuracy: 98.44% 12 Validation loss: 0.096400 Best loss: 0.052236 Accuracy: 98.20% 13 Validation loss: 0.048862 Best loss: 0.048862 Accuracy: 99.18% 14 Validation loss: 0.065636 Best loss: 0.048862 Accuracy: 98.87% 15 Validation loss: 0.090576 Best loss: 0.048862 Accuracy: 98.71% 16 Validation loss: 0.077621 Best loss: 0.048862 Accuracy: 98.75% 17 Validation loss: 0.074449 Best loss: 0.048862 Accuracy: 98.55% 18 Validation loss: 0.061563 Best loss: 0.048862 Accuracy: 98.87% 19 Validation loss: 0.064389 Best loss: 0.048862 Accuracy: 98.79% 20 Validation loss: 0.043998 Best loss: 0.043998 Accuracy: 99.10% 21 Validation loss: 0.055705 Best loss: 0.043998 Accuracy: 99.06% 22 Validation loss: 0.063131 Best loss: 0.043998 Accuracy: 98.98% 23 Validation loss: 0.055159 Best loss: 0.043998 Accuracy: 99.06% 24 Validation loss: 0.065498 Best loss: 0.043998 Accuracy: 98.98% 25 Validation loss: 0.091555 Best loss: 0.043998 Accuracy: 98.79% 26 Validation loss: 0.083490 Best loss: 0.043998 Accuracy: 98.75% 27 Validation loss: 0.056418 Best loss: 0.043998 Accuracy: 98.94% 28 Validation loss: 0.040654 Best loss: 0.040654 Accuracy: 99.06% 29 Validation loss: 0.051193 Best loss: 0.040654 Accuracy: 98.79% 30 Validation loss: 0.058877 Best loss: 0.040654 Accuracy: 98.67% 31 Validation loss: 0.055916 Best loss: 0.040654 Accuracy: 99.02% 32 Validation loss: 0.045816 Best loss: 0.040654 Accuracy: 99.06% 33 Validation loss: 0.052880 Best loss: 0.040654 Accuracy: 99.14% 34 Validation loss: 0.057122 Best loss: 0.040654 Accuracy: 98.98% 35 Validation loss: 0.061393 Best loss: 0.040654 Accuracy: 98.87% 36 Validation loss: 0.048490 Best loss: 0.040654 Accuracy: 99.02% 37 Validation loss: 0.055549 Best loss: 0.040654 Accuracy: 99.10% 38 Validation loss: 0.056459 Best loss: 0.040654 Accuracy: 98.94% 39 Validation loss: 0.045904 Best loss: 0.040654 Accuracy: 99.34% 40 Validation loss: 0.052483 Best loss: 0.040654 Accuracy: 99.10% 41 Validation loss: 0.059005 Best loss: 0.040654 Accuracy: 99.02% 42 Validation loss: 0.060041 Best loss: 0.040654 Accuracy: 98.91% 43 Validation loss: 0.062492 Best loss: 0.040654 Accuracy: 98.98% 44 Validation loss: 0.059222 Best loss: 0.040654 Accuracy: 99.14% 45 Validation loss: 0.056598 Best loss: 0.040654 Accuracy: 98.83% 46 Validation loss: 0.060366 Best loss: 0.040654 Accuracy: 98.98% 47 Validation loss: 0.051070 Best loss: 0.040654 Accuracy: 99.10% 48 Validation loss: 0.050885 Best loss: 0.040654 Accuracy: 99.06% 49 Validation loss: 0.081166 Best loss: 0.040654 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=50, batch_norm_momentum=0.999, learning_rate=0.01, total= 2.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 1.802614 Best loss: 1.802614 Accuracy: 92.26% 1 Validation loss: 0.680390 Best loss: 0.680390 Accuracy: 94.41% 2 Validation loss: 0.294159 Best loss: 0.294159 Accuracy: 96.05% 3 Validation loss: 0.311890 Best loss: 0.294159 Accuracy: 96.17% 4 Validation loss: 0.166064 Best loss: 0.166064 Accuracy: 96.91% 5 Validation loss: 0.244669 Best loss: 0.166064 Accuracy: 95.70% 6 Validation loss: 0.160455 Best loss: 0.160455 Accuracy: 96.48% 7 Validation loss: 0.225204 Best loss: 0.160455 Accuracy: 94.84% 8 Validation loss: 0.244384 Best loss: 0.160455 Accuracy: 95.19% 9 Validation loss: 0.246485 Best loss: 0.160455 Accuracy: 95.23% 10 Validation loss: 0.197342 Best loss: 0.160455 Accuracy: 96.21% 11 Validation loss: 0.218079 Best loss: 0.160455 Accuracy: 96.29% 12 Validation loss: 0.173121 Best loss: 0.160455 Accuracy: 96.83% 13 Validation loss: 0.192037 Best loss: 0.160455 Accuracy: 96.60% 14 Validation loss: 0.296363 Best loss: 0.160455 Accuracy: 95.50% 15 Validation loss: 0.163530 Best loss: 0.160455 Accuracy: 97.15% 16 Validation loss: 0.126829 Best loss: 0.126829 Accuracy: 97.50% 17 Validation loss: 0.159128 Best loss: 0.126829 Accuracy: 97.19% 18 Validation loss: 0.105223 Best loss: 0.105223 Accuracy: 98.01% 19 Validation loss: 0.134105 Best loss: 0.105223 Accuracy: 97.34% 20 Validation loss: 0.204274 Best loss: 0.105223 Accuracy: 96.68% 21 Validation loss: 0.217637 Best loss: 0.105223 Accuracy: 96.44% 22 Validation loss: 0.188687 Best loss: 0.105223 Accuracy: 96.68% 23 Validation loss: 0.337875 Best loss: 0.105223 Accuracy: 95.70% 24 Validation loss: 0.162147 Best loss: 0.105223 Accuracy: 97.69% 25 Validation loss: 0.237319 Best loss: 0.105223 Accuracy: 96.91% 26 Validation loss: 0.167855 Best loss: 0.105223 Accuracy: 97.15% 27 Validation loss: 0.231626 Best loss: 0.105223 Accuracy: 96.87% 28 Validation loss: 0.259213 Best loss: 0.105223 Accuracy: 96.72% 29 Validation loss: 0.223539 Best loss: 0.105223 Accuracy: 97.30% 30 Validation loss: 0.139092 Best loss: 0.105223 Accuracy: 97.65% 31 Validation loss: 0.124882 Best loss: 0.105223 Accuracy: 97.69% 32 Validation loss: 0.202693 Best loss: 0.105223 Accuracy: 96.87% 33 Validation loss: 0.123389 Best loss: 0.105223 Accuracy: 97.97% 34 Validation loss: 0.205928 Best loss: 0.105223 Accuracy: 97.42% 35 Validation loss: 0.216608 Best loss: 0.105223 Accuracy: 97.26% 36 Validation loss: 0.248719 Best loss: 0.105223 Accuracy: 97.07% 37 Validation loss: 0.214557 Best loss: 0.105223 Accuracy: 97.19% 38 Validation loss: 0.198377 Best loss: 0.105223 Accuracy: 97.58% 39 Validation loss: 0.305196 Best loss: 0.105223 Accuracy: 96.29% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1, total= 14.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 5.165814 Best loss: 5.165814 Accuracy: 73.89% 1 Validation loss: 2.058486 Best loss: 2.058486 Accuracy: 86.98% 2 Validation loss: 0.434613 Best loss: 0.434613 Accuracy: 92.89% 3 Validation loss: 0.178704 Best loss: 0.178704 Accuracy: 95.82% 4 Validation loss: 0.213401 Best loss: 0.178704 Accuracy: 95.78% 5 Validation loss: 0.197452 Best loss: 0.178704 Accuracy: 96.17% 6 Validation loss: 0.165688 Best loss: 0.165688 Accuracy: 96.21% 7 Validation loss: 0.151866 Best loss: 0.151866 Accuracy: 96.99% 8 Validation loss: 0.108531 Best loss: 0.108531 Accuracy: 97.54% 9 Validation loss: 0.157622 Best loss: 0.108531 Accuracy: 96.44% 10 Validation loss: 0.126449 Best loss: 0.108531 Accuracy: 97.38% 11 Validation loss: 0.174741 Best loss: 0.108531 Accuracy: 96.95% 12 Validation loss: 0.130402 Best loss: 0.108531 Accuracy: 97.65% 13 Validation loss: 0.141532 Best loss: 0.108531 Accuracy: 96.95% 14 Validation loss: 0.182577 Best loss: 0.108531 Accuracy: 96.48% 15 Validation loss: 0.183603 Best loss: 0.108531 Accuracy: 96.64% 16 Validation loss: 0.110791 Best loss: 0.108531 Accuracy: 97.81% 17 Validation loss: 0.157344 Best loss: 0.108531 Accuracy: 96.87% 18 Validation loss: 0.204774 Best loss: 0.108531 Accuracy: 96.76% 19 Validation loss: 0.185657 Best loss: 0.108531 Accuracy: 97.07% 20 Validation loss: 0.181923 Best loss: 0.108531 Accuracy: 96.44% 21 Validation loss: 0.129342 Best loss: 0.108531 Accuracy: 97.62% 22 Validation loss: 0.146727 Best loss: 0.108531 Accuracy: 97.54% 23 Validation loss: 0.133516 Best loss: 0.108531 Accuracy: 97.50% 24 Validation loss: 0.134343 Best loss: 0.108531 Accuracy: 97.07% 25 Validation loss: 0.097471 Best loss: 0.097471 Accuracy: 97.97% 26 Validation loss: 0.107231 Best loss: 0.097471 Accuracy: 97.85% 27 Validation loss: 0.134751 Best loss: 0.097471 Accuracy: 97.19% 28 Validation loss: 0.109842 Best loss: 0.097471 Accuracy: 97.85% 29 Validation loss: 0.122841 Best loss: 0.097471 Accuracy: 97.58% 30 Validation loss: 0.112380 Best loss: 0.097471 Accuracy: 97.50% 31 Validation loss: 0.157046 Best loss: 0.097471 Accuracy: 97.65% 32 Validation loss: 0.155459 Best loss: 0.097471 Accuracy: 97.11% 33 Validation loss: 0.152077 Best loss: 0.097471 Accuracy: 97.62% 34 Validation loss: 0.131897 Best loss: 0.097471 Accuracy: 97.26% 35 Validation loss: 0.152686 Best loss: 0.097471 Accuracy: 96.95% 36 Validation loss: 0.130338 Best loss: 0.097471 Accuracy: 97.58% 37 Validation loss: 0.109003 Best loss: 0.097471 Accuracy: 97.97% 38 Validation loss: 0.126668 Best loss: 0.097471 Accuracy: 97.65% 39 Validation loss: 0.115442 Best loss: 0.097471 Accuracy: 97.58% 40 Validation loss: 0.179789 Best loss: 0.097471 Accuracy: 96.83% 41 Validation loss: 0.122781 Best loss: 0.097471 Accuracy: 97.62% 42 Validation loss: 0.122425 Best loss: 0.097471 Accuracy: 97.65% 43 Validation loss: 0.219854 Best loss: 0.097471 Accuracy: 96.44% 44 Validation loss: 0.125541 Best loss: 0.097471 Accuracy: 97.54% 45 Validation loss: 0.119769 Best loss: 0.097471 Accuracy: 97.50% 46 Validation loss: 0.194818 Best loss: 0.097471 Accuracy: 96.33% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1, total= 17.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 8.264340 Best loss: 8.264340 Accuracy: 71.97% 1 Validation loss: 2.503082 Best loss: 2.503082 Accuracy: 84.01% 2 Validation loss: 1.015884 Best loss: 1.015884 Accuracy: 88.66% 3 Validation loss: 0.677211 Best loss: 0.677211 Accuracy: 90.19% 4 Validation loss: 0.326623 Best loss: 0.326623 Accuracy: 94.37% 5 Validation loss: 0.640489 Best loss: 0.326623 Accuracy: 92.57% 6 Validation loss: 0.226022 Best loss: 0.226022 Accuracy: 95.86% 7 Validation loss: 0.225128 Best loss: 0.225128 Accuracy: 96.29% 8 Validation loss: 0.141858 Best loss: 0.141858 Accuracy: 97.22% 9 Validation loss: 0.138471 Best loss: 0.138471 Accuracy: 96.76% 10 Validation loss: 0.125028 Best loss: 0.125028 Accuracy: 97.38% 11 Validation loss: 0.196872 Best loss: 0.125028 Accuracy: 96.25% 12 Validation loss: 0.139832 Best loss: 0.125028 Accuracy: 96.95% 13 Validation loss: 0.169196 Best loss: 0.125028 Accuracy: 96.68% 14 Validation loss: 0.130380 Best loss: 0.125028 Accuracy: 97.07% 15 Validation loss: 0.273393 Best loss: 0.125028 Accuracy: 94.72% 16 Validation loss: 0.152223 Best loss: 0.125028 Accuracy: 97.58% 17 Validation loss: 0.116613 Best loss: 0.116613 Accuracy: 97.69% 18 Validation loss: 0.202519 Best loss: 0.116613 Accuracy: 96.72% 19 Validation loss: 0.170793 Best loss: 0.116613 Accuracy: 97.03% 20 Validation loss: 0.186850 Best loss: 0.116613 Accuracy: 96.87% 21 Validation loss: 0.144545 Best loss: 0.116613 Accuracy: 96.99% 22 Validation loss: 0.147123 Best loss: 0.116613 Accuracy: 97.73% 23 Validation loss: 0.130117 Best loss: 0.116613 Accuracy: 97.50% 24 Validation loss: 0.128018 Best loss: 0.116613 Accuracy: 97.54% 25 Validation loss: 0.139703 Best loss: 0.116613 Accuracy: 97.58% 26 Validation loss: 0.149486 Best loss: 0.116613 Accuracy: 97.42% 27 Validation loss: 0.120759 Best loss: 0.116613 Accuracy: 97.73% 28 Validation loss: 0.139312 Best loss: 0.116613 Accuracy: 97.69% 29 Validation loss: 0.138380 Best loss: 0.116613 Accuracy: 97.30% 30 Validation loss: 0.126808 Best loss: 0.116613 Accuracy: 97.73% 31 Validation loss: 0.144378 Best loss: 0.116613 Accuracy: 97.26% 32 Validation loss: 0.290613 Best loss: 0.116613 Accuracy: 93.67% 33 Validation loss: 0.158954 Best loss: 0.116613 Accuracy: 97.54% 34 Validation loss: 0.135659 Best loss: 0.116613 Accuracy: 97.38% 35 Validation loss: 0.111026 Best loss: 0.111026 Accuracy: 97.89% 36 Validation loss: 0.172330 Best loss: 0.111026 Accuracy: 96.91% 37 Validation loss: 0.141775 Best loss: 0.111026 Accuracy: 97.38% 38 Validation loss: 0.139581 Best loss: 0.111026 Accuracy: 97.62% 39 Validation loss: 0.150113 Best loss: 0.111026 Accuracy: 97.38% 40 Validation loss: 0.119903 Best loss: 0.111026 Accuracy: 97.89% 41 Validation loss: 0.142538 Best loss: 0.111026 Accuracy: 97.65% 42 Validation loss: 0.146955 Best loss: 0.111026 Accuracy: 97.85% 43 Validation loss: 0.140973 Best loss: 0.111026 Accuracy: 97.77% 44 Validation loss: 0.137171 Best loss: 0.111026 Accuracy: 97.81% 45 Validation loss: 0.195574 Best loss: 0.111026 Accuracy: 96.87% 46 Validation loss: 0.158188 Best loss: 0.111026 Accuracy: 97.62% 47 Validation loss: 0.122647 Best loss: 0.111026 Accuracy: 97.81% 48 Validation loss: 0.129478 Best loss: 0.111026 Accuracy: 97.81% 49 Validation loss: 0.171597 Best loss: 0.111026 Accuracy: 97.38% 50 Validation loss: 0.156874 Best loss: 0.111026 Accuracy: 96.79% 51 Validation loss: 0.152896 Best loss: 0.111026 Accuracy: 97.58% 52 Validation loss: 0.126636 Best loss: 0.111026 Accuracy: 98.01% 53 Validation loss: 0.162269 Best loss: 0.111026 Accuracy: 97.42% 54 Validation loss: 0.128079 Best loss: 0.111026 Accuracy: 97.89% 55 Validation loss: 0.135787 Best loss: 0.111026 Accuracy: 97.97% 56 Validation loss: 0.156677 Best loss: 0.111026 Accuracy: 97.62% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.1, total= 19.2s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.089562 Best loss: 0.089562 Accuracy: 97.62% 1 Validation loss: 0.099691 Best loss: 0.089562 Accuracy: 97.73% 2 Validation loss: 0.056249 Best loss: 0.056249 Accuracy: 98.67% 3 Validation loss: 0.048598 Best loss: 0.048598 Accuracy: 98.63% 4 Validation loss: 0.038399 Best loss: 0.038399 Accuracy: 98.94% 5 Validation loss: 0.043839 Best loss: 0.038399 Accuracy: 98.75% 6 Validation loss: 0.038919 Best loss: 0.038399 Accuracy: 98.83% 7 Validation loss: 0.051833 Best loss: 0.038399 Accuracy: 98.67% 8 Validation loss: 0.038252 Best loss: 0.038252 Accuracy: 99.06% 9 Validation loss: 0.055630 Best loss: 0.038252 Accuracy: 98.83% 10 Validation loss: 0.069801 Best loss: 0.038252 Accuracy: 98.67% 11 Validation loss: 0.047470 Best loss: 0.038252 Accuracy: 98.79% 12 Validation loss: 0.050089 Best loss: 0.038252 Accuracy: 98.79% 13 Validation loss: 0.041374 Best loss: 0.038252 Accuracy: 99.10% 14 Validation loss: 0.032406 Best loss: 0.032406 Accuracy: 99.02% 15 Validation loss: 0.054509 Best loss: 0.032406 Accuracy: 98.67% 16 Validation loss: 0.040613 Best loss: 0.032406 Accuracy: 99.06% 17 Validation loss: 0.055707 Best loss: 0.032406 Accuracy: 98.83% 18 Validation loss: 0.042059 Best loss: 0.032406 Accuracy: 99.10% 19 Validation loss: 0.044712 Best loss: 0.032406 Accuracy: 98.75% 20 Validation loss: 0.047190 Best loss: 0.032406 Accuracy: 98.94% 21 Validation loss: 0.054088 Best loss: 0.032406 Accuracy: 98.63% 22 Validation loss: 0.034966 Best loss: 0.032406 Accuracy: 99.06% 23 Validation loss: 0.049625 Best loss: 0.032406 Accuracy: 98.98% 24 Validation loss: 0.042628 Best loss: 0.032406 Accuracy: 99.10% 25 Validation loss: 0.063609 Best loss: 0.032406 Accuracy: 98.83% 26 Validation loss: 0.046528 Best loss: 0.032406 Accuracy: 99.10% 27 Validation loss: 0.037443 Best loss: 0.032406 Accuracy: 99.10% 28 Validation loss: 0.054400 Best loss: 0.032406 Accuracy: 99.14% 29 Validation loss: 0.042370 Best loss: 0.032406 Accuracy: 99.22% 30 Validation loss: 0.062299 Best loss: 0.032406 Accuracy: 98.79% 31 Validation loss: 0.059551 Best loss: 0.032406 Accuracy: 98.67% 32 Validation loss: 0.045498 Best loss: 0.032406 Accuracy: 98.98% 33 Validation loss: 0.050378 Best loss: 0.032406 Accuracy: 98.98% 34 Validation loss: 0.053859 Best loss: 0.032406 Accuracy: 98.94% 35 Validation loss: 0.064649 Best loss: 0.032406 Accuracy: 98.71% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 46.0s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.079169 Best loss: 0.079169 Accuracy: 97.89% 1 Validation loss: 0.069839 Best loss: 0.069839 Accuracy: 98.08% 2 Validation loss: 0.050104 Best loss: 0.050104 Accuracy: 98.63% 3 Validation loss: 0.051268 Best loss: 0.050104 Accuracy: 98.36% 4 Validation loss: 0.063797 Best loss: 0.050104 Accuracy: 98.67% 5 Validation loss: 0.040553 Best loss: 0.040553 Accuracy: 99.02% 6 Validation loss: 0.041229 Best loss: 0.040553 Accuracy: 99.02% 7 Validation loss: 0.039160 Best loss: 0.039160 Accuracy: 98.91% 8 Validation loss: 0.051602 Best loss: 0.039160 Accuracy: 98.83% 9 Validation loss: 0.043171 Best loss: 0.039160 Accuracy: 98.98% 10 Validation loss: 0.052268 Best loss: 0.039160 Accuracy: 98.67% 11 Validation loss: 0.041593 Best loss: 0.039160 Accuracy: 98.87% 12 Validation loss: 0.038145 Best loss: 0.038145 Accuracy: 98.83% 13 Validation loss: 0.041357 Best loss: 0.038145 Accuracy: 98.75% 14 Validation loss: 0.038458 Best loss: 0.038145 Accuracy: 99.02% 15 Validation loss: 0.051196 Best loss: 0.038145 Accuracy: 98.79% 16 Validation loss: 0.034041 Best loss: 0.034041 Accuracy: 99.02% 17 Validation loss: 0.044020 Best loss: 0.034041 Accuracy: 98.91% 18 Validation loss: 0.040581 Best loss: 0.034041 Accuracy: 98.98% 19 Validation loss: 0.038737 Best loss: 0.034041 Accuracy: 98.87% 20 Validation loss: 0.052951 Best loss: 0.034041 Accuracy: 98.91% 21 Validation loss: 0.044145 Best loss: 0.034041 Accuracy: 99.06% 22 Validation loss: 0.058458 Best loss: 0.034041 Accuracy: 98.75% 23 Validation loss: 0.044081 Best loss: 0.034041 Accuracy: 99.10% 24 Validation loss: 0.039248 Best loss: 0.034041 Accuracy: 99.22% 25 Validation loss: 0.048226 Best loss: 0.034041 Accuracy: 98.98% 26 Validation loss: 0.039696 Best loss: 0.034041 Accuracy: 98.91% 27 Validation loss: 0.042700 Best loss: 0.034041 Accuracy: 98.94% 28 Validation loss: 0.050258 Best loss: 0.034041 Accuracy: 98.98% 29 Validation loss: 0.036507 Best loss: 0.034041 Accuracy: 99.02% 30 Validation loss: 0.039601 Best loss: 0.034041 Accuracy: 99.02% 31 Validation loss: 0.032075 Best loss: 0.032075 Accuracy: 99.14% 32 Validation loss: 0.033708 Best loss: 0.032075 Accuracy: 99.30% 33 Validation loss: 0.053287 Best loss: 0.032075 Accuracy: 98.79% 34 Validation loss: 0.058925 Best loss: 0.032075 Accuracy: 99.02% 35 Validation loss: 0.037754 Best loss: 0.032075 Accuracy: 99.18% 36 Validation loss: 0.060017 Best loss: 0.032075 Accuracy: 98.75% 37 Validation loss: 0.029369 Best loss: 0.029369 Accuracy: 99.22% 38 Validation loss: 0.041681 Best loss: 0.029369 Accuracy: 99.14% 39 Validation loss: 0.039317 Best loss: 0.029369 Accuracy: 99.26% 40 Validation loss: 0.039426 Best loss: 0.029369 Accuracy: 99.14% 41 Validation loss: 0.038493 Best loss: 0.029369 Accuracy: 99.26% 42 Validation loss: 0.045820 Best loss: 0.029369 Accuracy: 99.02% 43 Validation loss: 0.055685 Best loss: 0.029369 Accuracy: 99.02% 44 Validation loss: 0.037622 Best loss: 0.029369 Accuracy: 99.26% 45 Validation loss: 0.059275 Best loss: 0.029369 Accuracy: 99.06% 46 Validation loss: 0.059914 Best loss: 0.029369 Accuracy: 98.75% 47 Validation loss: 0.060695 Best loss: 0.029369 Accuracy: 99.26% 48 Validation loss: 0.047443 Best loss: 0.029369 Accuracy: 99.26% 49 Validation loss: 0.060666 Best loss: 0.029369 Accuracy: 98.98% 50 Validation loss: 0.053808 Best loss: 0.029369 Accuracy: 99.10% 51 Validation loss: 0.040235 Best loss: 0.029369 Accuracy: 99.22% 52 Validation loss: 0.041169 Best loss: 0.029369 Accuracy: 99.18% 53 Validation loss: 0.048914 Best loss: 0.029369 Accuracy: 99.10% 54 Validation loss: 0.050571 Best loss: 0.029369 Accuracy: 99.22% 55 Validation loss: 0.041067 Best loss: 0.029369 Accuracy: 99.30% 56 Validation loss: 0.035789 Best loss: 0.029369 Accuracy: 99.34% 57 Validation loss: 0.065277 Best loss: 0.029369 Accuracy: 98.94% 58 Validation loss: 0.046772 Best loss: 0.029369 Accuracy: 99.14% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 1.3min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.076095 Best loss: 0.076095 Accuracy: 98.08% 1 Validation loss: 0.083436 Best loss: 0.076095 Accuracy: 97.65% 2 Validation loss: 0.039430 Best loss: 0.039430 Accuracy: 98.67% 3 Validation loss: 0.056619 Best loss: 0.039430 Accuracy: 98.59% 4 Validation loss: 0.050400 Best loss: 0.039430 Accuracy: 98.67% 5 Validation loss: 0.044779 Best loss: 0.039430 Accuracy: 98.91% 6 Validation loss: 0.038474 Best loss: 0.038474 Accuracy: 98.83% 7 Validation loss: 0.050784 Best loss: 0.038474 Accuracy: 98.75% 8 Validation loss: 0.054795 Best loss: 0.038474 Accuracy: 98.71% 9 Validation loss: 0.045632 Best loss: 0.038474 Accuracy: 98.48% 10 Validation loss: 0.044239 Best loss: 0.038474 Accuracy: 98.75% 11 Validation loss: 0.047149 Best loss: 0.038474 Accuracy: 98.79% 12 Validation loss: 0.051998 Best loss: 0.038474 Accuracy: 98.94% 13 Validation loss: 0.035945 Best loss: 0.035945 Accuracy: 98.87% 14 Validation loss: 0.045893 Best loss: 0.035945 Accuracy: 98.87% 15 Validation loss: 0.038574 Best loss: 0.035945 Accuracy: 99.14% 16 Validation loss: 0.041970 Best loss: 0.035945 Accuracy: 98.98% 17 Validation loss: 0.064473 Best loss: 0.035945 Accuracy: 98.71% 18 Validation loss: 0.033813 Best loss: 0.033813 Accuracy: 99.22% 19 Validation loss: 0.062043 Best loss: 0.033813 Accuracy: 98.71% 20 Validation loss: 0.048532 Best loss: 0.033813 Accuracy: 98.71% 21 Validation loss: 0.033300 Best loss: 0.033300 Accuracy: 99.14% 22 Validation loss: 0.060661 Best loss: 0.033300 Accuracy: 98.83% 23 Validation loss: 0.051267 Best loss: 0.033300 Accuracy: 98.87% 24 Validation loss: 0.032985 Best loss: 0.032985 Accuracy: 99.18% 25 Validation loss: 0.043730 Best loss: 0.032985 Accuracy: 99.14% 26 Validation loss: 0.038901 Best loss: 0.032985 Accuracy: 99.02% 27 Validation loss: 0.074131 Best loss: 0.032985 Accuracy: 98.32% 28 Validation loss: 0.035406 Best loss: 0.032985 Accuracy: 99.37% 29 Validation loss: 0.066857 Best loss: 0.032985 Accuracy: 98.44% 30 Validation loss: 0.043690 Best loss: 0.032985 Accuracy: 99.26% 31 Validation loss: 0.033457 Best loss: 0.032985 Accuracy: 99.22% 32 Validation loss: 0.060961 Best loss: 0.032985 Accuracy: 98.75% 33 Validation loss: 0.044913 Best loss: 0.032985 Accuracy: 99.10% 34 Validation loss: 0.029916 Best loss: 0.029916 Accuracy: 99.34% 35 Validation loss: 0.058271 Best loss: 0.029916 Accuracy: 98.94% 36 Validation loss: 0.046300 Best loss: 0.029916 Accuracy: 99.18% 37 Validation loss: 0.051773 Best loss: 0.029916 Accuracy: 98.98% 38 Validation loss: 0.026250 Best loss: 0.026250 Accuracy: 99.22% 39 Validation loss: 0.054722 Best loss: 0.026250 Accuracy: 99.14% 40 Validation loss: 0.022853 Best loss: 0.022853 Accuracy: 99.30% 41 Validation loss: 0.030382 Best loss: 0.022853 Accuracy: 99.41% 42 Validation loss: 0.029724 Best loss: 0.022853 Accuracy: 99.26% 43 Validation loss: 0.081969 Best loss: 0.022853 Accuracy: 98.75% 44 Validation loss: 0.042476 Best loss: 0.022853 Accuracy: 99.02% 45 Validation loss: 0.028882 Best loss: 0.022853 Accuracy: 99.37% 46 Validation loss: 0.043540 Best loss: 0.022853 Accuracy: 99.02% 47 Validation loss: 0.045187 Best loss: 0.022853 Accuracy: 99.22% 48 Validation loss: 0.030881 Best loss: 0.022853 Accuracy: 99.22% 49 Validation loss: 0.063200 Best loss: 0.022853 Accuracy: 98.98% 50 Validation loss: 0.035576 Best loss: 0.022853 Accuracy: 99.30% 51 Validation loss: 0.026472 Best loss: 0.022853 Accuracy: 99.18% 52 Validation loss: 0.026976 Best loss: 0.022853 Accuracy: 99.22% 53 Validation loss: 0.031402 Best loss: 0.022853 Accuracy: 99.30% 54 Validation loss: 0.040936 Best loss: 0.022853 Accuracy: 99.22% 55 Validation loss: 0.029381 Best loss: 0.022853 Accuracy: 99.37% 56 Validation loss: 0.040076 Best loss: 0.022853 Accuracy: 99.18% 57 Validation loss: 0.039740 Best loss: 0.022853 Accuracy: 99.14% 58 Validation loss: 0.065127 Best loss: 0.022853 Accuracy: 98.63% 59 Validation loss: 0.028401 Best loss: 0.022853 Accuracy: 99.37% 60 Validation loss: 0.034977 Best loss: 0.022853 Accuracy: 99.30% 61 Validation loss: 0.031884 Best loss: 0.022853 Accuracy: 99.14% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1 0 Validation loss: 0.191237 Best loss: 0.191237 Accuracy: 94.96% 1 Validation loss: 0.092097 Best loss: 0.092097 Accuracy: 97.46% 2 Validation loss: 0.133243 Best loss: 0.092097 Accuracy: 96.83% 3 Validation loss: 0.100522 Best loss: 0.092097 Accuracy: 96.25% 4 Validation loss: 0.112462 Best loss: 0.092097 Accuracy: 96.87% 5 Validation loss: 0.148533 Best loss: 0.092097 Accuracy: 96.87% 6 Validation loss: 0.042695 Best loss: 0.042695 Accuracy: 98.91% 7 Validation loss: 0.075638 Best loss: 0.042695 Accuracy: 97.62% 8 Validation loss: 0.218445 Best loss: 0.042695 Accuracy: 96.60% 9 Validation loss: 0.077283 Best loss: 0.042695 Accuracy: 98.24% 10 Validation loss: 0.056739 Best loss: 0.042695 Accuracy: 98.48% 11 Validation loss: 0.074576 Best loss: 0.042695 Accuracy: 98.20% 12 Validation loss: 0.118723 Best loss: 0.042695 Accuracy: 98.01% 13 Validation loss: 0.053117 Best loss: 0.042695 Accuracy: 98.28% 14 Validation loss: 0.068111 Best loss: 0.042695 Accuracy: 98.44% 15 Validation loss: 0.060562 Best loss: 0.042695 Accuracy: 98.40% 16 Validation loss: 0.077181 Best loss: 0.042695 Accuracy: 98.20% 17 Validation loss: 0.082865 Best loss: 0.042695 Accuracy: 98.20% 18 Validation loss: 0.243991 Best loss: 0.042695 Accuracy: 96.91% 19 Validation loss: 0.069151 Best loss: 0.042695 Accuracy: 98.51% 20 Validation loss: 0.074824 Best loss: 0.042695 Accuracy: 98.40% 21 Validation loss: 0.064319 Best loss: 0.042695 Accuracy: 97.97% 22 Validation loss: 0.076168 Best loss: 0.042695 Accuracy: 98.20% 23 Validation loss: 0.120039 Best loss: 0.042695 Accuracy: 97.93% 24 Validation loss: 0.090815 Best loss: 0.042695 Accuracy: 98.16% 25 Validation loss: 0.088011 Best loss: 0.042695 Accuracy: 98.36% 26 Validation loss: 0.066743 Best loss: 0.042695 Accuracy: 98.63% 27 Validation loss: 0.075012 Best loss: 0.042695 Accuracy: 98.48% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1, total= 5.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1 0 Validation loss: 0.152922 Best loss: 0.152922 Accuracy: 95.54% 1 Validation loss: 0.235430 Best loss: 0.152922 Accuracy: 93.71% 2 Validation loss: 0.158619 Best loss: 0.152922 Accuracy: 95.31% 3 Validation loss: 0.078993 Best loss: 0.078993 Accuracy: 97.81% 4 Validation loss: 0.065490 Best loss: 0.065490 Accuracy: 97.93% 5 Validation loss: 0.334119 Best loss: 0.065490 Accuracy: 95.11% 6 Validation loss: 0.103983 Best loss: 0.065490 Accuracy: 97.38% 7 Validation loss: 0.072090 Best loss: 0.065490 Accuracy: 98.16% 8 Validation loss: 0.083960 Best loss: 0.065490 Accuracy: 98.36% 9 Validation loss: 0.061834 Best loss: 0.061834 Accuracy: 98.20% 10 Validation loss: 0.059492 Best loss: 0.059492 Accuracy: 98.32% 11 Validation loss: 0.079601 Best loss: 0.059492 Accuracy: 98.01% 12 Validation loss: 0.078385 Best loss: 0.059492 Accuracy: 97.89% 13 Validation loss: 0.076013 Best loss: 0.059492 Accuracy: 97.93% 14 Validation loss: 0.066531 Best loss: 0.059492 Accuracy: 98.51% 15 Validation loss: 0.068362 Best loss: 0.059492 Accuracy: 98.28% 16 Validation loss: 0.157399 Best loss: 0.059492 Accuracy: 96.21% 17 Validation loss: 0.069517 Best loss: 0.059492 Accuracy: 98.36% 18 Validation loss: 0.072621 Best loss: 0.059492 Accuracy: 98.20% 19 Validation loss: 0.073968 Best loss: 0.059492 Accuracy: 98.40% 20 Validation loss: 0.051567 Best loss: 0.051567 Accuracy: 98.55% 21 Validation loss: 0.061100 Best loss: 0.051567 Accuracy: 98.51% 22 Validation loss: 0.121555 Best loss: 0.051567 Accuracy: 97.11% 23 Validation loss: 0.067434 Best loss: 0.051567 Accuracy: 98.24% 24 Validation loss: 0.056729 Best loss: 0.051567 Accuracy: 98.63% 25 Validation loss: 0.058594 Best loss: 0.051567 Accuracy: 98.12% 26 Validation loss: 0.061723 Best loss: 0.051567 Accuracy: 98.67% 27 Validation loss: 0.095544 Best loss: 0.051567 Accuracy: 97.65% 28 Validation loss: 0.064707 Best loss: 0.051567 Accuracy: 98.59% 29 Validation loss: 0.045553 Best loss: 0.045553 Accuracy: 98.71% 30 Validation loss: 0.053162 Best loss: 0.045553 Accuracy: 98.83% 31 Validation loss: 0.058948 Best loss: 0.045553 Accuracy: 98.48% 32 Validation loss: 0.069783 Best loss: 0.045553 Accuracy: 98.51% 33 Validation loss: 0.056007 Best loss: 0.045553 Accuracy: 98.59% 34 Validation loss: 0.107107 Best loss: 0.045553 Accuracy: 98.08% 35 Validation loss: 0.053041 Best loss: 0.045553 Accuracy: 98.75% 36 Validation loss: 0.075278 Best loss: 0.045553 Accuracy: 98.28% 37 Validation loss: 0.105700 Best loss: 0.045553 Accuracy: 97.73% 38 Validation loss: 0.063821 Best loss: 0.045553 Accuracy: 98.75% 39 Validation loss: 0.075803 Best loss: 0.045553 Accuracy: 98.40% 40 Validation loss: 0.049310 Best loss: 0.045553 Accuracy: 98.79% 41 Validation loss: 0.062334 Best loss: 0.045553 Accuracy: 98.59% 42 Validation loss: 0.065781 Best loss: 0.045553 Accuracy: 98.51% 43 Validation loss: 0.080956 Best loss: 0.045553 Accuracy: 98.44% 44 Validation loss: 0.080660 Best loss: 0.045553 Accuracy: 98.67% 45 Validation loss: 0.078019 Best loss: 0.045553 Accuracy: 98.36% 46 Validation loss: 0.083383 Best loss: 0.045553 Accuracy: 98.75% 47 Validation loss: 0.087813 Best loss: 0.045553 Accuracy: 98.91% 48 Validation loss: 0.300110 Best loss: 0.045553 Accuracy: 95.78% 49 Validation loss: 0.063171 Best loss: 0.045553 Accuracy: 98.91% 50 Validation loss: 0.089799 Best loss: 0.045553 Accuracy: 98.51% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1, total= 9.6min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1 0 Validation loss: 0.158824 Best loss: 0.158824 Accuracy: 94.61% 1 Validation loss: 0.156168 Best loss: 0.156168 Accuracy: 94.53% 2 Validation loss: 0.103413 Best loss: 0.103413 Accuracy: 96.60% 3 Validation loss: 0.079545 Best loss: 0.079545 Accuracy: 97.81% 4 Validation loss: 0.135134 Best loss: 0.079545 Accuracy: 95.78% 5 Validation loss: 0.172773 Best loss: 0.079545 Accuracy: 95.47% 6 Validation loss: 0.083104 Best loss: 0.079545 Accuracy: 97.30% 7 Validation loss: 0.071045 Best loss: 0.071045 Accuracy: 97.97% 8 Validation loss: 0.071601 Best loss: 0.071045 Accuracy: 98.51% 9 Validation loss: 0.071301 Best loss: 0.071045 Accuracy: 97.85% 10 Validation loss: 0.062445 Best loss: 0.062445 Accuracy: 98.16% 11 Validation loss: 0.075321 Best loss: 0.062445 Accuracy: 97.81% 12 Validation loss: 0.069213 Best loss: 0.062445 Accuracy: 98.12% 13 Validation loss: 0.119064 Best loss: 0.062445 Accuracy: 97.77% 14 Validation loss: 0.053385 Best loss: 0.053385 Accuracy: 98.67% 15 Validation loss: 0.068230 Best loss: 0.053385 Accuracy: 98.55% 16 Validation loss: 0.060359 Best loss: 0.053385 Accuracy: 98.48% 17 Validation loss: 0.067233 Best loss: 0.053385 Accuracy: 98.44% 18 Validation loss: 0.066739 Best loss: 0.053385 Accuracy: 98.32% 19 Validation loss: 0.051754 Best loss: 0.051754 Accuracy: 98.63% 20 Validation loss: 0.043889 Best loss: 0.043889 Accuracy: 98.98% 21 Validation loss: 0.054645 Best loss: 0.043889 Accuracy: 98.79% 22 Validation loss: 0.068452 Best loss: 0.043889 Accuracy: 98.55% 23 Validation loss: 0.042846 Best loss: 0.042846 Accuracy: 98.75% 24 Validation loss: 0.104264 Best loss: 0.042846 Accuracy: 97.85% 25 Validation loss: 0.105397 Best loss: 0.042846 Accuracy: 97.58% 26 Validation loss: 0.060144 Best loss: 0.042846 Accuracy: 98.94% 27 Validation loss: 0.196754 Best loss: 0.042846 Accuracy: 94.49% 28 Validation loss: 0.093156 Best loss: 0.042846 Accuracy: 98.28% 29 Validation loss: 0.114105 Best loss: 0.042846 Accuracy: 98.67% 30 Validation loss: 0.071281 Best loss: 0.042846 Accuracy: 98.48% 31 Validation loss: 0.289312 Best loss: 0.042846 Accuracy: 91.40% 32 Validation loss: 0.058953 Best loss: 0.042846 Accuracy: 98.91% 33 Validation loss: 0.053666 Best loss: 0.042846 Accuracy: 98.98% 34 Validation loss: 0.100407 Best loss: 0.042846 Accuracy: 98.71% 35 Validation loss: 0.103268 Best loss: 0.042846 Accuracy: 98.28% 36 Validation loss: 0.071982 Best loss: 0.042846 Accuracy: 98.71% 37 Validation loss: 0.068736 Best loss: 0.042846 Accuracy: 98.71% 38 Validation loss: 0.098347 Best loss: 0.042846 Accuracy: 98.40% 39 Validation loss: 0.062048 Best loss: 0.042846 Accuracy: 98.40% 40 Validation loss: 0.046456 Best loss: 0.042846 Accuracy: 98.83% 41 Validation loss: 0.076481 Best loss: 0.042846 Accuracy: 98.08% 42 Validation loss: 0.066822 Best loss: 0.042846 Accuracy: 98.75% 43 Validation loss: 0.050351 Best loss: 0.042846 Accuracy: 98.71% 44 Validation loss: 0.050065 Best loss: 0.042846 Accuracy: 98.63% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.1, total= 8.5min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.096097 Best loss: 0.096097 Accuracy: 96.95% 1 Validation loss: 0.074899 Best loss: 0.074899 Accuracy: 97.77% 2 Validation loss: 0.061224 Best loss: 0.061224 Accuracy: 97.93% 3 Validation loss: 0.056629 Best loss: 0.056629 Accuracy: 98.28% 4 Validation loss: 0.091907 Best loss: 0.056629 Accuracy: 97.11% 5 Validation loss: 0.047967 Best loss: 0.047967 Accuracy: 98.79% 6 Validation loss: 0.049487 Best loss: 0.047967 Accuracy: 98.63% 7 Validation loss: 0.054900 Best loss: 0.047967 Accuracy: 98.24% 8 Validation loss: 0.073632 Best loss: 0.047967 Accuracy: 98.20% 9 Validation loss: 0.055707 Best loss: 0.047967 Accuracy: 98.63% 10 Validation loss: 0.049217 Best loss: 0.047967 Accuracy: 98.63% 11 Validation loss: 0.042169 Best loss: 0.042169 Accuracy: 98.83% 12 Validation loss: 0.052475 Best loss: 0.042169 Accuracy: 98.83% 13 Validation loss: 0.053142 Best loss: 0.042169 Accuracy: 98.75% 14 Validation loss: 0.065256 Best loss: 0.042169 Accuracy: 98.44% 15 Validation loss: 0.053648 Best loss: 0.042169 Accuracy: 98.79% 16 Validation loss: 0.041947 Best loss: 0.041947 Accuracy: 98.79% 17 Validation loss: 0.043588 Best loss: 0.041947 Accuracy: 99.02% 18 Validation loss: 0.045085 Best loss: 0.041947 Accuracy: 99.02% 19 Validation loss: 0.069981 Best loss: 0.041947 Accuracy: 98.75% 20 Validation loss: 0.059038 Best loss: 0.041947 Accuracy: 98.67% 21 Validation loss: 0.052233 Best loss: 0.041947 Accuracy: 98.91% 22 Validation loss: 0.051311 Best loss: 0.041947 Accuracy: 98.63% 23 Validation loss: 0.039596 Best loss: 0.039596 Accuracy: 98.94% 24 Validation loss: 0.056459 Best loss: 0.039596 Accuracy: 98.79% 25 Validation loss: 0.058109 Best loss: 0.039596 Accuracy: 98.91% 26 Validation loss: 0.046119 Best loss: 0.039596 Accuracy: 99.06% 27 Validation loss: 0.178447 Best loss: 0.039596 Accuracy: 96.64% 28 Validation loss: 0.291258 Best loss: 0.039596 Accuracy: 97.19% 29 Validation loss: 0.045635 Best loss: 0.039596 Accuracy: 99.26% 30 Validation loss: 0.053196 Best loss: 0.039596 Accuracy: 99.02% 31 Validation loss: 0.035705 Best loss: 0.035705 Accuracy: 99.26% 32 Validation loss: 0.045334 Best loss: 0.035705 Accuracy: 98.98% 33 Validation loss: 0.041382 Best loss: 0.035705 Accuracy: 99.22% 34 Validation loss: 0.044672 Best loss: 0.035705 Accuracy: 99.30% 35 Validation loss: 0.043010 Best loss: 0.035705 Accuracy: 99.06% 36 Validation loss: 0.060424 Best loss: 0.035705 Accuracy: 98.79% 37 Validation loss: 0.057870 Best loss: 0.035705 Accuracy: 99.06% 38 Validation loss: 0.054574 Best loss: 0.035705 Accuracy: 99.14% 39 Validation loss: 0.065027 Best loss: 0.035705 Accuracy: 98.87% 40 Validation loss: 0.083726 Best loss: 0.035705 Accuracy: 98.55% 41 Validation loss: 0.068665 Best loss: 0.035705 Accuracy: 98.71% 42 Validation loss: 0.097839 Best loss: 0.035705 Accuracy: 98.83% 43 Validation loss: 0.073917 Best loss: 0.035705 Accuracy: 98.79% 44 Validation loss: 0.079722 Best loss: 0.035705 Accuracy: 98.83% 45 Validation loss: 0.091944 Best loss: 0.035705 Accuracy: 98.36% 46 Validation loss: 0.069913 Best loss: 0.035705 Accuracy: 99.02% 47 Validation loss: 0.069113 Best loss: 0.035705 Accuracy: 98.98% 48 Validation loss: 0.085614 Best loss: 0.035705 Accuracy: 98.79% 49 Validation loss: 0.114392 Best loss: 0.035705 Accuracy: 98.67% 50 Validation loss: 0.157583 Best loss: 0.035705 Accuracy: 98.79% 51 Validation loss: 0.081185 Best loss: 0.035705 Accuracy: 98.91% 52 Validation loss: 0.081546 Best loss: 0.035705 Accuracy: 98.75% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 1.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.166863 Best loss: 0.166863 Accuracy: 95.23% 1 Validation loss: 0.063224 Best loss: 0.063224 Accuracy: 98.08% 2 Validation loss: 0.066103 Best loss: 0.063224 Accuracy: 97.97% 3 Validation loss: 0.061465 Best loss: 0.061465 Accuracy: 98.20% 4 Validation loss: 0.098393 Best loss: 0.061465 Accuracy: 97.38% 5 Validation loss: 0.065593 Best loss: 0.061465 Accuracy: 98.12% 6 Validation loss: 0.068421 Best loss: 0.061465 Accuracy: 98.32% 7 Validation loss: 0.054731 Best loss: 0.054731 Accuracy: 98.59% 8 Validation loss: 0.059369 Best loss: 0.054731 Accuracy: 98.63% 9 Validation loss: 0.051146 Best loss: 0.051146 Accuracy: 98.59% 10 Validation loss: 0.060011 Best loss: 0.051146 Accuracy: 98.63% 11 Validation loss: 0.061836 Best loss: 0.051146 Accuracy: 98.20% 12 Validation loss: 0.070623 Best loss: 0.051146 Accuracy: 98.28% 13 Validation loss: 0.066495 Best loss: 0.051146 Accuracy: 98.48% 14 Validation loss: 0.046406 Best loss: 0.046406 Accuracy: 98.51% 15 Validation loss: 0.047332 Best loss: 0.046406 Accuracy: 98.79% 16 Validation loss: 0.041074 Best loss: 0.041074 Accuracy: 98.98% 17 Validation loss: 0.082180 Best loss: 0.041074 Accuracy: 98.51% 18 Validation loss: 0.058820 Best loss: 0.041074 Accuracy: 98.75% 19 Validation loss: 0.060253 Best loss: 0.041074 Accuracy: 98.63% 20 Validation loss: 0.059298 Best loss: 0.041074 Accuracy: 98.67% 21 Validation loss: 0.103008 Best loss: 0.041074 Accuracy: 97.77% 22 Validation loss: 0.278972 Best loss: 0.041074 Accuracy: 96.48% 23 Validation loss: 0.067768 Best loss: 0.041074 Accuracy: 98.67% 24 Validation loss: 0.041303 Best loss: 0.041074 Accuracy: 98.94% 25 Validation loss: 0.052461 Best loss: 0.041074 Accuracy: 98.79% 26 Validation loss: 0.041306 Best loss: 0.041074 Accuracy: 99.18% 27 Validation loss: 0.051640 Best loss: 0.041074 Accuracy: 99.02% 28 Validation loss: 0.055914 Best loss: 0.041074 Accuracy: 98.91% 29 Validation loss: 0.055958 Best loss: 0.041074 Accuracy: 98.67% 30 Validation loss: 0.115773 Best loss: 0.041074 Accuracy: 98.55% 31 Validation loss: 0.047722 Best loss: 0.041074 Accuracy: 99.02% 32 Validation loss: 0.046479 Best loss: 0.041074 Accuracy: 98.98% 33 Validation loss: 0.052498 Best loss: 0.041074 Accuracy: 98.91% 34 Validation loss: 0.073987 Best loss: 0.041074 Accuracy: 98.83% 35 Validation loss: 0.066219 Best loss: 0.041074 Accuracy: 98.75% 36 Validation loss: 0.079405 Best loss: 0.041074 Accuracy: 98.91% 37 Validation loss: 0.357208 Best loss: 0.041074 Accuracy: 97.81% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 49.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.082513 Best loss: 0.082513 Accuracy: 97.65% 1 Validation loss: 0.066309 Best loss: 0.066309 Accuracy: 97.89% 2 Validation loss: 0.061049 Best loss: 0.061049 Accuracy: 98.05% 3 Validation loss: 0.050600 Best loss: 0.050600 Accuracy: 98.40% 4 Validation loss: 0.050369 Best loss: 0.050369 Accuracy: 98.51% 5 Validation loss: 0.039875 Best loss: 0.039875 Accuracy: 98.67% 6 Validation loss: 0.044277 Best loss: 0.039875 Accuracy: 98.55% 7 Validation loss: 0.056779 Best loss: 0.039875 Accuracy: 98.51% 8 Validation loss: 0.047049 Best loss: 0.039875 Accuracy: 98.71% 9 Validation loss: 0.048402 Best loss: 0.039875 Accuracy: 98.55% 10 Validation loss: 0.046535 Best loss: 0.039875 Accuracy: 98.83% 11 Validation loss: 0.059279 Best loss: 0.039875 Accuracy: 98.59% 12 Validation loss: 0.054254 Best loss: 0.039875 Accuracy: 98.75% 13 Validation loss: 0.044835 Best loss: 0.039875 Accuracy: 98.91% 14 Validation loss: 0.058531 Best loss: 0.039875 Accuracy: 98.87% 15 Validation loss: 0.047711 Best loss: 0.039875 Accuracy: 98.71% 16 Validation loss: 0.053535 Best loss: 0.039875 Accuracy: 99.18% 17 Validation loss: 0.064144 Best loss: 0.039875 Accuracy: 98.67% 18 Validation loss: 0.045292 Best loss: 0.039875 Accuracy: 98.87% 19 Validation loss: 0.056056 Best loss: 0.039875 Accuracy: 98.87% 20 Validation loss: 0.095829 Best loss: 0.039875 Accuracy: 98.94% 21 Validation loss: 0.124700 Best loss: 0.039875 Accuracy: 98.51% 22 Validation loss: 0.038030 Best loss: 0.038030 Accuracy: 99.14% 23 Validation loss: 0.036242 Best loss: 0.036242 Accuracy: 99.22% 24 Validation loss: 0.029601 Best loss: 0.029601 Accuracy: 99.22% 25 Validation loss: 0.041481 Best loss: 0.029601 Accuracy: 99.22% 26 Validation loss: 0.035303 Best loss: 0.029601 Accuracy: 99.37% 27 Validation loss: 0.047202 Best loss: 0.029601 Accuracy: 98.91% 28 Validation loss: 0.047763 Best loss: 0.029601 Accuracy: 99.02% 29 Validation loss: 0.063396 Best loss: 0.029601 Accuracy: 98.75% 30 Validation loss: 0.050096 Best loss: 0.029601 Accuracy: 99.06% 31 Validation loss: 0.047902 Best loss: 0.029601 Accuracy: 98.98% 32 Validation loss: 0.073094 Best loss: 0.029601 Accuracy: 98.79% 33 Validation loss: 0.064006 Best loss: 0.029601 Accuracy: 98.59% 34 Validation loss: 0.096572 Best loss: 0.029601 Accuracy: 98.75% 35 Validation loss: 0.092028 Best loss: 0.029601 Accuracy: 98.55% 36 Validation loss: 0.041048 Best loss: 0.029601 Accuracy: 99.18% 37 Validation loss: 0.035247 Best loss: 0.029601 Accuracy: 99.34% 38 Validation loss: 0.044975 Best loss: 0.029601 Accuracy: 99.10% 39 Validation loss: 0.037285 Best loss: 0.029601 Accuracy: 99.22% 40 Validation loss: 0.051610 Best loss: 0.029601 Accuracy: 99.02% 41 Validation loss: 0.046031 Best loss: 0.029601 Accuracy: 99.14% 42 Validation loss: 0.035414 Best loss: 0.029601 Accuracy: 99.41% 43 Validation loss: 0.067801 Best loss: 0.029601 Accuracy: 99.10% 44 Validation loss: 0.057790 Best loss: 0.029601 Accuracy: 99.10% 45 Validation loss: 0.112993 Best loss: 0.029601 Accuracy: 98.16% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.05, total= 56.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.119953 Best loss: 0.119953 Accuracy: 96.36% 1 Validation loss: 0.078522 Best loss: 0.078522 Accuracy: 97.81% 2 Validation loss: 0.075696 Best loss: 0.075696 Accuracy: 98.12% 3 Validation loss: 0.062940 Best loss: 0.062940 Accuracy: 98.08% 4 Validation loss: 0.060841 Best loss: 0.060841 Accuracy: 98.51% 5 Validation loss: 0.056713 Best loss: 0.056713 Accuracy: 98.32% 6 Validation loss: 0.064660 Best loss: 0.056713 Accuracy: 98.36% 7 Validation loss: 0.050055 Best loss: 0.050055 Accuracy: 98.83% 8 Validation loss: 0.037890 Best loss: 0.037890 Accuracy: 99.06% 9 Validation loss: 0.064065 Best loss: 0.037890 Accuracy: 98.48% 10 Validation loss: 0.089571 Best loss: 0.037890 Accuracy: 97.93% 11 Validation loss: 0.063107 Best loss: 0.037890 Accuracy: 98.36% 12 Validation loss: 0.113254 Best loss: 0.037890 Accuracy: 98.01% 13 Validation loss: 0.058948 Best loss: 0.037890 Accuracy: 98.55% 14 Validation loss: 0.069744 Best loss: 0.037890 Accuracy: 98.28% 15 Validation loss: 0.040086 Best loss: 0.037890 Accuracy: 98.79% 16 Validation loss: 0.043180 Best loss: 0.037890 Accuracy: 99.10% 17 Validation loss: 0.064249 Best loss: 0.037890 Accuracy: 98.63% 18 Validation loss: 0.102954 Best loss: 0.037890 Accuracy: 98.40% 19 Validation loss: 0.054006 Best loss: 0.037890 Accuracy: 98.98% 20 Validation loss: 0.066613 Best loss: 0.037890 Accuracy: 98.91% 21 Validation loss: 0.066150 Best loss: 0.037890 Accuracy: 98.91% 22 Validation loss: 0.051992 Best loss: 0.037890 Accuracy: 98.87% 23 Validation loss: 0.075611 Best loss: 0.037890 Accuracy: 98.71% 24 Validation loss: 0.061837 Best loss: 0.037890 Accuracy: 98.79% 25 Validation loss: 0.080415 Best loss: 0.037890 Accuracy: 98.55% 26 Validation loss: 0.045645 Best loss: 0.037890 Accuracy: 99.10% 27 Validation loss: 0.087932 Best loss: 0.037890 Accuracy: 98.75% 28 Validation loss: 0.136102 Best loss: 0.037890 Accuracy: 98.48% 29 Validation loss: 0.082168 Best loss: 0.037890 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05, total= 1.1min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.118905 Best loss: 0.118905 Accuracy: 96.60% 1 Validation loss: 0.064978 Best loss: 0.064978 Accuracy: 98.24% 2 Validation loss: 0.076262 Best loss: 0.064978 Accuracy: 97.73% 3 Validation loss: 0.069101 Best loss: 0.064978 Accuracy: 98.12% 4 Validation loss: 0.061641 Best loss: 0.061641 Accuracy: 98.12% 5 Validation loss: 0.075714 Best loss: 0.061641 Accuracy: 98.24% 6 Validation loss: 0.066860 Best loss: 0.061641 Accuracy: 98.24% 7 Validation loss: 0.056727 Best loss: 0.056727 Accuracy: 98.24% 8 Validation loss: 0.055266 Best loss: 0.055266 Accuracy: 98.55% 9 Validation loss: 0.079627 Best loss: 0.055266 Accuracy: 98.51% 10 Validation loss: 0.049672 Best loss: 0.049672 Accuracy: 98.63% 11 Validation loss: 0.052443 Best loss: 0.049672 Accuracy: 98.67% 12 Validation loss: 0.033579 Best loss: 0.033579 Accuracy: 98.94% 13 Validation loss: 0.056108 Best loss: 0.033579 Accuracy: 98.28% 14 Validation loss: 0.056456 Best loss: 0.033579 Accuracy: 98.67% 15 Validation loss: 0.072289 Best loss: 0.033579 Accuracy: 98.28% 16 Validation loss: 0.090885 Best loss: 0.033579 Accuracy: 98.44% 17 Validation loss: 0.091550 Best loss: 0.033579 Accuracy: 98.24% 18 Validation loss: 0.058170 Best loss: 0.033579 Accuracy: 98.79% 19 Validation loss: 0.048338 Best loss: 0.033579 Accuracy: 98.94% 20 Validation loss: 0.061145 Best loss: 0.033579 Accuracy: 98.83% 21 Validation loss: 0.076846 Best loss: 0.033579 Accuracy: 98.67% 22 Validation loss: 0.220808 Best loss: 0.033579 Accuracy: 97.54% 23 Validation loss: 0.060453 Best loss: 0.033579 Accuracy: 98.83% 24 Validation loss: 0.051474 Best loss: 0.033579 Accuracy: 99.06% 25 Validation loss: 0.060004 Best loss: 0.033579 Accuracy: 99.06% 26 Validation loss: 0.068576 Best loss: 0.033579 Accuracy: 98.91% 27 Validation loss: 0.066004 Best loss: 0.033579 Accuracy: 98.79% 28 Validation loss: 0.059230 Best loss: 0.033579 Accuracy: 99.02% 29 Validation loss: 0.064178 Best loss: 0.033579 Accuracy: 99.02% 30 Validation loss: 0.083687 Best loss: 0.033579 Accuracy: 98.87% 31 Validation loss: 0.259885 Best loss: 0.033579 Accuracy: 97.54% 32 Validation loss: 0.078671 Best loss: 0.033579 Accuracy: 98.87% 33 Validation loss: 0.058562 Best loss: 0.033579 Accuracy: 98.94% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05, total= 1.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05 0 Validation loss: 0.134531 Best loss: 0.134531 Accuracy: 96.36% 1 Validation loss: 0.078223 Best loss: 0.078223 Accuracy: 97.77% 2 Validation loss: 0.064522 Best loss: 0.064522 Accuracy: 97.69% 3 Validation loss: 0.060874 Best loss: 0.060874 Accuracy: 98.24% 4 Validation loss: 0.070387 Best loss: 0.060874 Accuracy: 98.08% 5 Validation loss: 0.055754 Best loss: 0.055754 Accuracy: 98.32% 6 Validation loss: 0.217674 Best loss: 0.055754 Accuracy: 94.61% 7 Validation loss: 0.069051 Best loss: 0.055754 Accuracy: 98.63% 8 Validation loss: 0.039916 Best loss: 0.039916 Accuracy: 98.83% 9 Validation loss: 0.071528 Best loss: 0.039916 Accuracy: 98.12% 10 Validation loss: 0.098026 Best loss: 0.039916 Accuracy: 98.16% 11 Validation loss: 0.035011 Best loss: 0.035011 Accuracy: 98.98% 12 Validation loss: 0.050572 Best loss: 0.035011 Accuracy: 98.83% 13 Validation loss: 0.044097 Best loss: 0.035011 Accuracy: 98.83% 14 Validation loss: 0.074217 Best loss: 0.035011 Accuracy: 98.40% 15 Validation loss: 0.114860 Best loss: 0.035011 Accuracy: 98.36% 16 Validation loss: 0.045512 Best loss: 0.035011 Accuracy: 98.83% 17 Validation loss: 0.041144 Best loss: 0.035011 Accuracy: 99.06% 18 Validation loss: 0.049665 Best loss: 0.035011 Accuracy: 98.75% 19 Validation loss: 0.061934 Best loss: 0.035011 Accuracy: 98.83% 20 Validation loss: 0.100040 Best loss: 0.035011 Accuracy: 98.67% 21 Validation loss: 0.046249 Best loss: 0.035011 Accuracy: 98.98% 22 Validation loss: 0.043154 Best loss: 0.035011 Accuracy: 99.30% 23 Validation loss: 0.051518 Best loss: 0.035011 Accuracy: 98.94% 24 Validation loss: 0.055067 Best loss: 0.035011 Accuracy: 99.10% 25 Validation loss: 0.142826 Best loss: 0.035011 Accuracy: 98.08% 26 Validation loss: 0.076284 Best loss: 0.035011 Accuracy: 98.91% 27 Validation loss: 0.058664 Best loss: 0.035011 Accuracy: 98.98% 28 Validation loss: 0.048520 Best loss: 0.035011 Accuracy: 99.22% 29 Validation loss: 0.052487 Best loss: 0.035011 Accuracy: 99.18% 30 Validation loss: 0.055717 Best loss: 0.035011 Accuracy: 99.30% 31 Validation loss: 0.079381 Best loss: 0.035011 Accuracy: 98.87% 32 Validation loss: 0.044947 Best loss: 0.035011 Accuracy: 99.26% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.9, learning_rate=0.05, total= 1.3min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.189957 Best loss: 0.189957 Accuracy: 96.48% 1 Validation loss: 0.160329 Best loss: 0.160329 Accuracy: 97.15% 2 Validation loss: 0.154407 Best loss: 0.154407 Accuracy: 96.83% 3 Validation loss: 0.151723 Best loss: 0.151723 Accuracy: 97.30% 4 Validation loss: 0.085206 Best loss: 0.085206 Accuracy: 98.12% 5 Validation loss: 0.062777 Best loss: 0.062777 Accuracy: 98.51% 6 Validation loss: 0.057991 Best loss: 0.057991 Accuracy: 98.79% 7 Validation loss: 0.129725 Best loss: 0.057991 Accuracy: 98.40% 8 Validation loss: 0.074284 Best loss: 0.057991 Accuracy: 98.71% 9 Validation loss: 0.079903 Best loss: 0.057991 Accuracy: 98.12% 10 Validation loss: 0.075492 Best loss: 0.057991 Accuracy: 98.51% 11 Validation loss: 0.080558 Best loss: 0.057991 Accuracy: 98.36% 12 Validation loss: 0.074840 Best loss: 0.057991 Accuracy: 98.67% 13 Validation loss: 0.088502 Best loss: 0.057991 Accuracy: 98.01% 14 Validation loss: 0.064204 Best loss: 0.057991 Accuracy: 98.83% 15 Validation loss: 0.065152 Best loss: 0.057991 Accuracy: 98.12% 16 Validation loss: 0.083179 Best loss: 0.057991 Accuracy: 98.28% 17 Validation loss: 0.059315 Best loss: 0.057991 Accuracy: 98.55% 18 Validation loss: 0.059511 Best loss: 0.057991 Accuracy: 98.79% 19 Validation loss: 0.054752 Best loss: 0.054752 Accuracy: 98.75% 20 Validation loss: 0.072097 Best loss: 0.054752 Accuracy: 98.51% 21 Validation loss: 0.061518 Best loss: 0.054752 Accuracy: 98.63% 22 Validation loss: 0.083093 Best loss: 0.054752 Accuracy: 98.48% 23 Validation loss: 0.086322 Best loss: 0.054752 Accuracy: 98.55% 24 Validation loss: 0.045265 Best loss: 0.045265 Accuracy: 98.98% 25 Validation loss: 0.060911 Best loss: 0.045265 Accuracy: 98.83% 26 Validation loss: 0.064055 Best loss: 0.045265 Accuracy: 98.71% 27 Validation loss: 0.081011 Best loss: 0.045265 Accuracy: 98.55% 28 Validation loss: 0.041492 Best loss: 0.041492 Accuracy: 98.91% 29 Validation loss: 0.077850 Best loss: 0.041492 Accuracy: 98.55% 30 Validation loss: 0.073632 Best loss: 0.041492 Accuracy: 98.55% 31 Validation loss: 0.094068 Best loss: 0.041492 Accuracy: 98.75% 32 Validation loss: 0.063347 Best loss: 0.041492 Accuracy: 98.98% 33 Validation loss: 0.066487 Best loss: 0.041492 Accuracy: 98.63% 34 Validation loss: 0.158368 Best loss: 0.041492 Accuracy: 98.36% 35 Validation loss: 0.067620 Best loss: 0.041492 Accuracy: 98.83% 36 Validation loss: 0.097271 Best loss: 0.041492 Accuracy: 98.16% 37 Validation loss: 0.070518 Best loss: 0.041492 Accuracy: 98.59% 38 Validation loss: 0.160563 Best loss: 0.041492 Accuracy: 98.28% 39 Validation loss: 0.065389 Best loss: 0.041492 Accuracy: 98.71% 40 Validation loss: 0.064383 Best loss: 0.041492 Accuracy: 98.94% 41 Validation loss: 0.071187 Best loss: 0.041492 Accuracy: 98.51% 42 Validation loss: 0.086017 Best loss: 0.041492 Accuracy: 98.20% 43 Validation loss: 0.053568 Best loss: 0.041492 Accuracy: 99.06% 44 Validation loss: 0.060282 Best loss: 0.041492 Accuracy: 98.94% 45 Validation loss: 0.084896 Best loss: 0.041492 Accuracy: 98.67% 46 Validation loss: 0.083269 Best loss: 0.041492 Accuracy: 98.71% 47 Validation loss: 0.077011 Best loss: 0.041492 Accuracy: 98.79% 48 Validation loss: 0.073016 Best loss: 0.041492 Accuracy: 98.75% 49 Validation loss: 0.072296 Best loss: 0.041492 Accuracy: 98.51% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.9min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.213420 Best loss: 0.213420 Accuracy: 94.84% 1 Validation loss: 0.134403 Best loss: 0.134403 Accuracy: 97.11% 2 Validation loss: 0.252159 Best loss: 0.134403 Accuracy: 95.66% 3 Validation loss: 0.104887 Best loss: 0.104887 Accuracy: 97.42% 4 Validation loss: 0.126352 Best loss: 0.104887 Accuracy: 97.65% 5 Validation loss: 0.061756 Best loss: 0.061756 Accuracy: 98.40% 6 Validation loss: 0.063004 Best loss: 0.061756 Accuracy: 98.51% 7 Validation loss: 0.066700 Best loss: 0.061756 Accuracy: 98.16% 8 Validation loss: 0.114466 Best loss: 0.061756 Accuracy: 98.05% 9 Validation loss: 0.063267 Best loss: 0.061756 Accuracy: 98.55% 10 Validation loss: 0.152120 Best loss: 0.061756 Accuracy: 97.11% 11 Validation loss: 0.064214 Best loss: 0.061756 Accuracy: 98.71% 12 Validation loss: 0.119430 Best loss: 0.061756 Accuracy: 97.93% 13 Validation loss: 0.072940 Best loss: 0.061756 Accuracy: 98.36% 14 Validation loss: 0.077974 Best loss: 0.061756 Accuracy: 98.16% 15 Validation loss: 1.482024 Best loss: 0.061756 Accuracy: 82.76% 16 Validation loss: 0.096573 Best loss: 0.061756 Accuracy: 97.89% 17 Validation loss: 0.073879 Best loss: 0.061756 Accuracy: 98.55% 18 Validation loss: 0.076976 Best loss: 0.061756 Accuracy: 98.75% 19 Validation loss: 0.083888 Best loss: 0.061756 Accuracy: 98.44% 20 Validation loss: 0.234781 Best loss: 0.061756 Accuracy: 97.15% 21 Validation loss: 0.068525 Best loss: 0.061756 Accuracy: 98.51% 22 Validation loss: 0.065245 Best loss: 0.061756 Accuracy: 98.87% 23 Validation loss: 0.072465 Best loss: 0.061756 Accuracy: 98.40% 24 Validation loss: 0.071496 Best loss: 0.061756 Accuracy: 98.67% 25 Validation loss: 0.075057 Best loss: 0.061756 Accuracy: 98.48% 26 Validation loss: 0.104478 Best loss: 0.061756 Accuracy: 98.20% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.1min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.568256 Best loss: 0.568256 Accuracy: 95.39% 1 Validation loss: 0.089134 Best loss: 0.089134 Accuracy: 97.73% 2 Validation loss: 0.054714 Best loss: 0.054714 Accuracy: 98.40% 3 Validation loss: 0.100418 Best loss: 0.054714 Accuracy: 97.69% 4 Validation loss: 0.075639 Best loss: 0.054714 Accuracy: 98.63% 5 Validation loss: 0.072489 Best loss: 0.054714 Accuracy: 98.24% 6 Validation loss: 0.053905 Best loss: 0.053905 Accuracy: 98.63% 7 Validation loss: 0.060333 Best loss: 0.053905 Accuracy: 98.71% 8 Validation loss: 0.060266 Best loss: 0.053905 Accuracy: 98.79% 9 Validation loss: 0.039129 Best loss: 0.039129 Accuracy: 98.91% 10 Validation loss: 0.067788 Best loss: 0.039129 Accuracy: 98.59% 11 Validation loss: 0.057442 Best loss: 0.039129 Accuracy: 98.55% 12 Validation loss: 0.096609 Best loss: 0.039129 Accuracy: 98.28% 13 Validation loss: 0.076343 Best loss: 0.039129 Accuracy: 98.01% 14 Validation loss: 0.095629 Best loss: 0.039129 Accuracy: 97.85% 15 Validation loss: 0.061509 Best loss: 0.039129 Accuracy: 98.79% 16 Validation loss: 0.053056 Best loss: 0.039129 Accuracy: 98.79% 17 Validation loss: 0.069083 Best loss: 0.039129 Accuracy: 98.51% 18 Validation loss: 0.069641 Best loss: 0.039129 Accuracy: 98.44% 19 Validation loss: 0.064215 Best loss: 0.039129 Accuracy: 98.71% 20 Validation loss: 0.049017 Best loss: 0.039129 Accuracy: 98.87% 21 Validation loss: 0.071751 Best loss: 0.039129 Accuracy: 98.79% 22 Validation loss: 0.050954 Best loss: 0.039129 Accuracy: 99.02% 23 Validation loss: 0.057134 Best loss: 0.039129 Accuracy: 98.63% 24 Validation loss: 0.060725 Best loss: 0.039129 Accuracy: 98.51% 25 Validation loss: 0.053063 Best loss: 0.039129 Accuracy: 98.94% 26 Validation loss: 0.147449 Best loss: 0.039129 Accuracy: 97.54% 27 Validation loss: 0.049764 Best loss: 0.039129 Accuracy: 98.83% 28 Validation loss: 0.094718 Best loss: 0.039129 Accuracy: 98.20% 29 Validation loss: 0.129856 Best loss: 0.039129 Accuracy: 98.36% 30 Validation loss: 0.046168 Best loss: 0.039129 Accuracy: 98.91% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05 0 Validation loss: 0.086532 Best loss: 0.086532 Accuracy: 97.50% 1 Validation loss: 0.067649 Best loss: 0.067649 Accuracy: 98.28% 2 Validation loss: 0.090312 Best loss: 0.067649 Accuracy: 97.62% 3 Validation loss: 0.063696 Best loss: 0.063696 Accuracy: 98.32% 4 Validation loss: 0.045145 Best loss: 0.045145 Accuracy: 98.91% 5 Validation loss: 0.052593 Best loss: 0.045145 Accuracy: 98.71% 6 Validation loss: 0.056327 Best loss: 0.045145 Accuracy: 98.32% 7 Validation loss: 0.053046 Best loss: 0.045145 Accuracy: 98.55% 8 Validation loss: 0.068647 Best loss: 0.045145 Accuracy: 98.32% 9 Validation loss: 0.052609 Best loss: 0.045145 Accuracy: 98.55% 10 Validation loss: 0.122311 Best loss: 0.045145 Accuracy: 97.38% 11 Validation loss: 0.058428 Best loss: 0.045145 Accuracy: 98.55% 12 Validation loss: 0.043696 Best loss: 0.043696 Accuracy: 98.91% 13 Validation loss: 0.055612 Best loss: 0.043696 Accuracy: 98.71% 14 Validation loss: 0.053481 Best loss: 0.043696 Accuracy: 98.75% 15 Validation loss: 0.072810 Best loss: 0.043696 Accuracy: 98.55% 16 Validation loss: 0.057101 Best loss: 0.043696 Accuracy: 99.02% 17 Validation loss: 0.045119 Best loss: 0.043696 Accuracy: 98.91% 18 Validation loss: 0.046615 Best loss: 0.043696 Accuracy: 98.91% 19 Validation loss: 0.065119 Best loss: 0.043696 Accuracy: 98.59% 20 Validation loss: 0.071097 Best loss: 0.043696 Accuracy: 98.79% 21 Validation loss: 0.063450 Best loss: 0.043696 Accuracy: 98.63% 22 Validation loss: 0.053859 Best loss: 0.043696 Accuracy: 98.91% 23 Validation loss: 0.066071 Best loss: 0.043696 Accuracy: 98.71% 24 Validation loss: 0.071458 Best loss: 0.043696 Accuracy: 98.48% 25 Validation loss: 0.054275 Best loss: 0.043696 Accuracy: 98.87% 26 Validation loss: 0.069325 Best loss: 0.043696 Accuracy: 98.55% 27 Validation loss: 0.052526 Best loss: 0.043696 Accuracy: 98.79% 28 Validation loss: 0.057033 Best loss: 0.043696 Accuracy: 98.79% 29 Validation loss: 0.062937 Best loss: 0.043696 Accuracy: 98.75% 30 Validation loss: 0.048814 Best loss: 0.043696 Accuracy: 99.02% 31 Validation loss: 0.070918 Best loss: 0.043696 Accuracy: 98.48% 32 Validation loss: 0.049730 Best loss: 0.043696 Accuracy: 98.87% 33 Validation loss: 0.049015 Best loss: 0.043696 Accuracy: 98.94% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05, total= 44.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05 0 Validation loss: 0.093574 Best loss: 0.093574 Accuracy: 97.15% 1 Validation loss: 0.052355 Best loss: 0.052355 Accuracy: 98.51% 2 Validation loss: 0.054674 Best loss: 0.052355 Accuracy: 98.32% 3 Validation loss: 0.056772 Best loss: 0.052355 Accuracy: 98.63% 4 Validation loss: 0.075534 Best loss: 0.052355 Accuracy: 98.24% 5 Validation loss: 0.044579 Best loss: 0.044579 Accuracy: 98.51% 6 Validation loss: 0.041037 Best loss: 0.041037 Accuracy: 98.63% 7 Validation loss: 0.063521 Best loss: 0.041037 Accuracy: 98.16% 8 Validation loss: 0.039852 Best loss: 0.039852 Accuracy: 99.02% 9 Validation loss: 0.050527 Best loss: 0.039852 Accuracy: 98.98% 10 Validation loss: 0.041459 Best loss: 0.039852 Accuracy: 98.91% 11 Validation loss: 0.030700 Best loss: 0.030700 Accuracy: 99.30% 12 Validation loss: 0.036250 Best loss: 0.030700 Accuracy: 99.18% 13 Validation loss: 0.047750 Best loss: 0.030700 Accuracy: 98.87% 14 Validation loss: 0.052537 Best loss: 0.030700 Accuracy: 98.32% 15 Validation loss: 0.059036 Best loss: 0.030700 Accuracy: 98.67% 16 Validation loss: 0.038907 Best loss: 0.030700 Accuracy: 98.71% 17 Validation loss: 0.049073 Best loss: 0.030700 Accuracy: 98.48% 18 Validation loss: 0.036036 Best loss: 0.030700 Accuracy: 99.14% 19 Validation loss: 0.059725 Best loss: 0.030700 Accuracy: 98.63% 20 Validation loss: 0.039481 Best loss: 0.030700 Accuracy: 98.87% 21 Validation loss: 0.052342 Best loss: 0.030700 Accuracy: 98.75% 22 Validation loss: 0.053810 Best loss: 0.030700 Accuracy: 98.94% 23 Validation loss: 0.073483 Best loss: 0.030700 Accuracy: 98.51% 24 Validation loss: 0.034999 Best loss: 0.030700 Accuracy: 98.87% 25 Validation loss: 0.054342 Best loss: 0.030700 Accuracy: 98.79% 26 Validation loss: 0.048526 Best loss: 0.030700 Accuracy: 98.87% 27 Validation loss: 0.050358 Best loss: 0.030700 Accuracy: 99.02% 28 Validation loss: 0.054618 Best loss: 0.030700 Accuracy: 98.67% 29 Validation loss: 0.038982 Best loss: 0.030700 Accuracy: 99.02% 30 Validation loss: 0.052903 Best loss: 0.030700 Accuracy: 98.91% 31 Validation loss: 0.041161 Best loss: 0.030700 Accuracy: 99.02% 32 Validation loss: 0.044792 Best loss: 0.030700 Accuracy: 98.83% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05, total= 42.0s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05 0 Validation loss: 0.093811 Best loss: 0.093811 Accuracy: 97.58% 1 Validation loss: 0.061405 Best loss: 0.061405 Accuracy: 98.40% 2 Validation loss: 0.065423 Best loss: 0.061405 Accuracy: 98.05% 3 Validation loss: 0.052224 Best loss: 0.052224 Accuracy: 98.40% 4 Validation loss: 0.054159 Best loss: 0.052224 Accuracy: 98.51% 5 Validation loss: 0.043437 Best loss: 0.043437 Accuracy: 98.75% 6 Validation loss: 0.044194 Best loss: 0.043437 Accuracy: 98.75% 7 Validation loss: 0.054837 Best loss: 0.043437 Accuracy: 98.83% 8 Validation loss: 0.062320 Best loss: 0.043437 Accuracy: 98.63% 9 Validation loss: 0.043942 Best loss: 0.043437 Accuracy: 98.67% 10 Validation loss: 0.041131 Best loss: 0.041131 Accuracy: 98.79% 11 Validation loss: 0.046901 Best loss: 0.041131 Accuracy: 98.98% 12 Validation loss: 0.060188 Best loss: 0.041131 Accuracy: 98.71% 13 Validation loss: 0.039344 Best loss: 0.039344 Accuracy: 98.91% 14 Validation loss: 0.036096 Best loss: 0.036096 Accuracy: 99.06% 15 Validation loss: 0.069457 Best loss: 0.036096 Accuracy: 98.71% 16 Validation loss: 0.048401 Best loss: 0.036096 Accuracy: 98.83% 17 Validation loss: 0.039044 Best loss: 0.036096 Accuracy: 98.87% 18 Validation loss: 0.037805 Best loss: 0.036096 Accuracy: 98.98% 19 Validation loss: 0.074021 Best loss: 0.036096 Accuracy: 98.44% 20 Validation loss: 0.045131 Best loss: 0.036096 Accuracy: 98.94% 21 Validation loss: 0.055489 Best loss: 0.036096 Accuracy: 98.94% 22 Validation loss: 0.046776 Best loss: 0.036096 Accuracy: 98.83% 23 Validation loss: 0.058806 Best loss: 0.036096 Accuracy: 98.63% 24 Validation loss: 0.043933 Best loss: 0.036096 Accuracy: 98.83% 25 Validation loss: 0.056751 Best loss: 0.036096 Accuracy: 98.71% 26 Validation loss: 0.040364 Best loss: 0.036096 Accuracy: 99.02% 27 Validation loss: 0.044120 Best loss: 0.036096 Accuracy: 98.83% 28 Validation loss: 0.044185 Best loss: 0.036096 Accuracy: 98.83% 29 Validation loss: 0.047170 Best loss: 0.036096 Accuracy: 99.06% 30 Validation loss: 0.046480 Best loss: 0.036096 Accuracy: 99.06% 31 Validation loss: 0.037640 Best loss: 0.036096 Accuracy: 99.26% 32 Validation loss: 0.043706 Best loss: 0.036096 Accuracy: 99.18% 33 Validation loss: 0.060873 Best loss: 0.036096 Accuracy: 99.06% 34 Validation loss: 0.078897 Best loss: 0.036096 Accuracy: 98.75% 35 Validation loss: 0.032735 Best loss: 0.032735 Accuracy: 99.14% 36 Validation loss: 0.036257 Best loss: 0.032735 Accuracy: 99.22% 37 Validation loss: 0.048007 Best loss: 0.032735 Accuracy: 99.06% 38 Validation loss: 0.031227 Best loss: 0.031227 Accuracy: 99.26% 39 Validation loss: 0.059010 Best loss: 0.031227 Accuracy: 98.91% 40 Validation loss: 0.070817 Best loss: 0.031227 Accuracy: 98.51% 41 Validation loss: 0.054356 Best loss: 0.031227 Accuracy: 98.75% 42 Validation loss: 0.042137 Best loss: 0.031227 Accuracy: 98.98% 43 Validation loss: 0.037502 Best loss: 0.031227 Accuracy: 99.10% 44 Validation loss: 0.053179 Best loss: 0.031227 Accuracy: 99.02% 45 Validation loss: 0.040623 Best loss: 0.031227 Accuracy: 99.22% 46 Validation loss: 0.050690 Best loss: 0.031227 Accuracy: 99.02% 47 Validation loss: 0.047941 Best loss: 0.031227 Accuracy: 99.22% 48 Validation loss: 0.039366 Best loss: 0.031227 Accuracy: 98.83% 49 Validation loss: 0.061373 Best loss: 0.031227 Accuracy: 98.79% 50 Validation loss: 0.042568 Best loss: 0.031227 Accuracy: 99.22% 51 Validation loss: 0.073682 Best loss: 0.031227 Accuracy: 98.75% 52 Validation loss: 0.046305 Best loss: 0.031227 Accuracy: 98.87% 53 Validation loss: 0.046521 Best loss: 0.031227 Accuracy: 99.18% 54 Validation loss: 0.040434 Best loss: 0.031227 Accuracy: 99.37% 55 Validation loss: 0.061188 Best loss: 0.031227 Accuracy: 99.26% 56 Validation loss: 0.045933 Best loss: 0.031227 Accuracy: 98.98% 57 Validation loss: 0.048979 Best loss: 0.031227 Accuracy: 98.87% 58 Validation loss: 0.048536 Best loss: 0.031227 Accuracy: 98.75% 59 Validation loss: 0.062792 Best loss: 0.031227 Accuracy: 99.14% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.05, total= 1.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.101361 Best loss: 0.101361 Accuracy: 97.38% 1 Validation loss: 0.056634 Best loss: 0.056634 Accuracy: 98.40% 2 Validation loss: 0.062733 Best loss: 0.056634 Accuracy: 98.32% 3 Validation loss: 0.047186 Best loss: 0.047186 Accuracy: 98.55% 4 Validation loss: 0.042384 Best loss: 0.042384 Accuracy: 98.79% 5 Validation loss: 0.058855 Best loss: 0.042384 Accuracy: 98.63% 6 Validation loss: 0.056469 Best loss: 0.042384 Accuracy: 98.44% 7 Validation loss: 0.045125 Best loss: 0.042384 Accuracy: 98.79% 8 Validation loss: 0.053539 Best loss: 0.042384 Accuracy: 98.59% 9 Validation loss: 0.046772 Best loss: 0.042384 Accuracy: 98.63% 10 Validation loss: 0.051513 Best loss: 0.042384 Accuracy: 98.71% 11 Validation loss: 0.051549 Best loss: 0.042384 Accuracy: 98.71% 12 Validation loss: 0.040723 Best loss: 0.040723 Accuracy: 98.79% 13 Validation loss: 0.042209 Best loss: 0.040723 Accuracy: 99.02% 14 Validation loss: 0.074009 Best loss: 0.040723 Accuracy: 98.32% 15 Validation loss: 0.046113 Best loss: 0.040723 Accuracy: 99.10% 16 Validation loss: 0.034630 Best loss: 0.034630 Accuracy: 99.02% 17 Validation loss: 0.041046 Best loss: 0.034630 Accuracy: 99.06% 18 Validation loss: 0.041666 Best loss: 0.034630 Accuracy: 98.91% 19 Validation loss: 0.038737 Best loss: 0.034630 Accuracy: 99.18% 20 Validation loss: 0.050184 Best loss: 0.034630 Accuracy: 98.51% 21 Validation loss: 0.050775 Best loss: 0.034630 Accuracy: 98.94% 22 Validation loss: 0.036956 Best loss: 0.034630 Accuracy: 99.06% 23 Validation loss: 0.037252 Best loss: 0.034630 Accuracy: 99.18% 24 Validation loss: 0.042159 Best loss: 0.034630 Accuracy: 99.02% 25 Validation loss: 0.045748 Best loss: 0.034630 Accuracy: 99.02% 26 Validation loss: 0.049942 Best loss: 0.034630 Accuracy: 99.06% 27 Validation loss: 0.031248 Best loss: 0.031248 Accuracy: 99.45% 28 Validation loss: 0.043546 Best loss: 0.031248 Accuracy: 98.83% 29 Validation loss: 0.051436 Best loss: 0.031248 Accuracy: 99.10% 30 Validation loss: 0.038956 Best loss: 0.031248 Accuracy: 99.10% 31 Validation loss: 0.036023 Best loss: 0.031248 Accuracy: 99.37% 32 Validation loss: 0.043975 Best loss: 0.031248 Accuracy: 99.06% 33 Validation loss: 0.037171 Best loss: 0.031248 Accuracy: 99.14% 34 Validation loss: 0.037125 Best loss: 0.031248 Accuracy: 99.26% 35 Validation loss: 0.027371 Best loss: 0.027371 Accuracy: 99.26% 36 Validation loss: 0.039045 Best loss: 0.027371 Accuracy: 99.26% 37 Validation loss: 0.051247 Best loss: 0.027371 Accuracy: 98.91% 38 Validation loss: 0.034193 Best loss: 0.027371 Accuracy: 99.22% 39 Validation loss: 0.053282 Best loss: 0.027371 Accuracy: 99.10% 40 Validation loss: 0.045184 Best loss: 0.027371 Accuracy: 99.22% 41 Validation loss: 0.050530 Best loss: 0.027371 Accuracy: 98.94% 42 Validation loss: 0.039375 Best loss: 0.027371 Accuracy: 99.26% 43 Validation loss: 0.031295 Best loss: 0.027371 Accuracy: 99.26% 44 Validation loss: 0.028525 Best loss: 0.027371 Accuracy: 99.41% 45 Validation loss: 0.045574 Best loss: 0.027371 Accuracy: 99.18% 46 Validation loss: 0.045027 Best loss: 0.027371 Accuracy: 99.06% 47 Validation loss: 0.039300 Best loss: 0.027371 Accuracy: 99.34% 48 Validation loss: 0.045775 Best loss: 0.027371 Accuracy: 99.02% 49 Validation loss: 0.028348 Best loss: 0.027371 Accuracy: 99.37% 50 Validation loss: 0.035397 Best loss: 0.027371 Accuracy: 99.41% 51 Validation loss: 0.033911 Best loss: 0.027371 Accuracy: 99.22% 52 Validation loss: 0.029011 Best loss: 0.027371 Accuracy: 99.34% 53 Validation loss: 0.040283 Best loss: 0.027371 Accuracy: 99.22% 54 Validation loss: 0.040629 Best loss: 0.027371 Accuracy: 99.22% 55 Validation loss: 0.049847 Best loss: 0.027371 Accuracy: 99.26% 56 Validation loss: 0.053520 Best loss: 0.027371 Accuracy: 99.22% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01, total= 2.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.092549 Best loss: 0.092549 Accuracy: 97.34% 1 Validation loss: 0.064571 Best loss: 0.064571 Accuracy: 98.01% 2 Validation loss: 0.059491 Best loss: 0.059491 Accuracy: 98.48% 3 Validation loss: 0.049967 Best loss: 0.049967 Accuracy: 98.51% 4 Validation loss: 0.059833 Best loss: 0.049967 Accuracy: 98.36% 5 Validation loss: 0.044000 Best loss: 0.044000 Accuracy: 98.55% 6 Validation loss: 0.043200 Best loss: 0.043200 Accuracy: 98.98% 7 Validation loss: 0.041788 Best loss: 0.041788 Accuracy: 98.79% 8 Validation loss: 0.038667 Best loss: 0.038667 Accuracy: 98.87% 9 Validation loss: 0.055702 Best loss: 0.038667 Accuracy: 98.44% 10 Validation loss: 0.055910 Best loss: 0.038667 Accuracy: 98.63% 11 Validation loss: 0.038992 Best loss: 0.038667 Accuracy: 99.02% 12 Validation loss: 0.027134 Best loss: 0.027134 Accuracy: 99.26% 13 Validation loss: 0.032221 Best loss: 0.027134 Accuracy: 99.22% 14 Validation loss: 0.034476 Best loss: 0.027134 Accuracy: 99.14% 15 Validation loss: 0.042527 Best loss: 0.027134 Accuracy: 98.83% 16 Validation loss: 0.051355 Best loss: 0.027134 Accuracy: 98.83% 17 Validation loss: 0.054261 Best loss: 0.027134 Accuracy: 98.91% 18 Validation loss: 0.042362 Best loss: 0.027134 Accuracy: 98.91% 19 Validation loss: 0.043199 Best loss: 0.027134 Accuracy: 98.94% 20 Validation loss: 0.047491 Best loss: 0.027134 Accuracy: 98.98% 21 Validation loss: 0.040319 Best loss: 0.027134 Accuracy: 98.91% 22 Validation loss: 0.099291 Best loss: 0.027134 Accuracy: 97.73% 23 Validation loss: 0.048962 Best loss: 0.027134 Accuracy: 98.87% 24 Validation loss: 0.049089 Best loss: 0.027134 Accuracy: 99.14% 25 Validation loss: 0.039527 Best loss: 0.027134 Accuracy: 99.30% 26 Validation loss: 0.041657 Best loss: 0.027134 Accuracy: 99.02% 27 Validation loss: 0.053018 Best loss: 0.027134 Accuracy: 98.98% 28 Validation loss: 0.040152 Best loss: 0.027134 Accuracy: 98.98% 29 Validation loss: 0.035304 Best loss: 0.027134 Accuracy: 99.22% 30 Validation loss: 0.037824 Best loss: 0.027134 Accuracy: 99.37% 31 Validation loss: 0.038888 Best loss: 0.027134 Accuracy: 99.30% 32 Validation loss: 0.042996 Best loss: 0.027134 Accuracy: 99.14% 33 Validation loss: 0.034324 Best loss: 0.027134 Accuracy: 99.06% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01, total= 1.3min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.071271 Best loss: 0.071271 Accuracy: 98.32% 1 Validation loss: 0.044345 Best loss: 0.044345 Accuracy: 98.51% 2 Validation loss: 0.072308 Best loss: 0.044345 Accuracy: 98.08% 3 Validation loss: 0.038101 Best loss: 0.038101 Accuracy: 98.94% 4 Validation loss: 0.048692 Best loss: 0.038101 Accuracy: 98.36% 5 Validation loss: 0.040603 Best loss: 0.038101 Accuracy: 98.79% 6 Validation loss: 0.033188 Best loss: 0.033188 Accuracy: 99.06% 7 Validation loss: 0.037999 Best loss: 0.033188 Accuracy: 98.91% 8 Validation loss: 0.036319 Best loss: 0.033188 Accuracy: 98.98% 9 Validation loss: 0.032673 Best loss: 0.032673 Accuracy: 99.06% 10 Validation loss: 0.037902 Best loss: 0.032673 Accuracy: 99.06% 11 Validation loss: 0.034692 Best loss: 0.032673 Accuracy: 98.94% 12 Validation loss: 0.048349 Best loss: 0.032673 Accuracy: 98.75% 13 Validation loss: 0.033729 Best loss: 0.032673 Accuracy: 99.34% 14 Validation loss: 0.026620 Best loss: 0.026620 Accuracy: 99.14% 15 Validation loss: 0.040350 Best loss: 0.026620 Accuracy: 98.94% 16 Validation loss: 0.025285 Best loss: 0.025285 Accuracy: 99.26% 17 Validation loss: 0.035274 Best loss: 0.025285 Accuracy: 99.06% 18 Validation loss: 0.029613 Best loss: 0.025285 Accuracy: 99.30% 19 Validation loss: 0.037800 Best loss: 0.025285 Accuracy: 99.10% 20 Validation loss: 0.034288 Best loss: 0.025285 Accuracy: 99.14% 21 Validation loss: 0.027491 Best loss: 0.025285 Accuracy: 99.26% 22 Validation loss: 0.032457 Best loss: 0.025285 Accuracy: 99.06% 23 Validation loss: 0.038170 Best loss: 0.025285 Accuracy: 99.14% 24 Validation loss: 0.033408 Best loss: 0.025285 Accuracy: 99.34% 25 Validation loss: 0.036800 Best loss: 0.025285 Accuracy: 99.18% 26 Validation loss: 0.030134 Best loss: 0.025285 Accuracy: 99.18% 27 Validation loss: 0.025687 Best loss: 0.025285 Accuracy: 99.18% 28 Validation loss: 0.044778 Best loss: 0.025285 Accuracy: 99.22% 29 Validation loss: 0.037268 Best loss: 0.025285 Accuracy: 99.22% 30 Validation loss: 0.033708 Best loss: 0.025285 Accuracy: 99.26% 31 Validation loss: 0.037357 Best loss: 0.025285 Accuracy: 99.22% 32 Validation loss: 0.040445 Best loss: 0.025285 Accuracy: 99.22% 33 Validation loss: 0.029714 Best loss: 0.025285 Accuracy: 99.30% 34 Validation loss: 0.055501 Best loss: 0.025285 Accuracy: 98.91% 35 Validation loss: 0.024715 Best loss: 0.024715 Accuracy: 99.26% 36 Validation loss: 0.021867 Best loss: 0.021867 Accuracy: 99.37% 37 Validation loss: 0.025029 Best loss: 0.021867 Accuracy: 99.26% 38 Validation loss: 0.033864 Best loss: 0.021867 Accuracy: 99.10% 39 Validation loss: 0.049888 Best loss: 0.021867 Accuracy: 98.94% 40 Validation loss: 0.036943 Best loss: 0.021867 Accuracy: 98.94% 41 Validation loss: 0.033773 Best loss: 0.021867 Accuracy: 99.10% 42 Validation loss: 0.034270 Best loss: 0.021867 Accuracy: 99.30% 43 Validation loss: 0.032282 Best loss: 0.021867 Accuracy: 99.37% 44 Validation loss: 0.040615 Best loss: 0.021867 Accuracy: 99.26% 45 Validation loss: 0.058749 Best loss: 0.021867 Accuracy: 99.06% 46 Validation loss: 0.031092 Best loss: 0.021867 Accuracy: 99.37% 47 Validation loss: 0.026929 Best loss: 0.021867 Accuracy: 99.26% 48 Validation loss: 0.038759 Best loss: 0.021867 Accuracy: 99.34% 49 Validation loss: 0.033030 Best loss: 0.021867 Accuracy: 99.22% 50 Validation loss: 0.027471 Best loss: 0.021867 Accuracy: 99.41% 51 Validation loss: 0.030415 Best loss: 0.021867 Accuracy: 99.41% 52 Validation loss: 0.034618 Best loss: 0.021867 Accuracy: 99.26% 53 Validation loss: 0.031653 Best loss: 0.021867 Accuracy: 99.34% 54 Validation loss: 0.039632 Best loss: 0.021867 Accuracy: 99.41% 55 Validation loss: 0.033564 Best loss: 0.021867 Accuracy: 99.26% 56 Validation loss: 0.056988 Best loss: 0.021867 Accuracy: 98.91% 57 Validation loss: 0.039476 Best loss: 0.021867 Accuracy: 99.22% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.01, total= 2.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.115120 Best loss: 0.115120 Accuracy: 96.17% 1 Validation loss: 0.075282 Best loss: 0.075282 Accuracy: 97.69% 2 Validation loss: 0.081064 Best loss: 0.075282 Accuracy: 97.42% 3 Validation loss: 0.080123 Best loss: 0.075282 Accuracy: 97.62% 4 Validation loss: 0.069102 Best loss: 0.069102 Accuracy: 97.81% 5 Validation loss: 0.078613 Best loss: 0.069102 Accuracy: 97.62% 6 Validation loss: 0.066066 Best loss: 0.066066 Accuracy: 97.81% 7 Validation loss: 0.075650 Best loss: 0.066066 Accuracy: 97.77% 8 Validation loss: 0.054160 Best loss: 0.054160 Accuracy: 98.55% 9 Validation loss: 0.052616 Best loss: 0.052616 Accuracy: 98.55% 10 Validation loss: 0.070449 Best loss: 0.052616 Accuracy: 98.28% 11 Validation loss: 0.101328 Best loss: 0.052616 Accuracy: 97.42% 12 Validation loss: 0.067146 Best loss: 0.052616 Accuracy: 97.97% 13 Validation loss: 0.054450 Best loss: 0.052616 Accuracy: 98.67% 14 Validation loss: 0.054754 Best loss: 0.052616 Accuracy: 98.67% 15 Validation loss: 0.060343 Best loss: 0.052616 Accuracy: 98.55% 16 Validation loss: 0.069446 Best loss: 0.052616 Accuracy: 98.32% 17 Validation loss: 0.070371 Best loss: 0.052616 Accuracy: 98.51% 18 Validation loss: 0.078650 Best loss: 0.052616 Accuracy: 98.16% 19 Validation loss: 0.088905 Best loss: 0.052616 Accuracy: 98.01% 20 Validation loss: 0.048011 Best loss: 0.048011 Accuracy: 98.51% 21 Validation loss: 0.054532 Best loss: 0.048011 Accuracy: 98.79% 22 Validation loss: 0.061423 Best loss: 0.048011 Accuracy: 98.59% 23 Validation loss: 0.065016 Best loss: 0.048011 Accuracy: 98.63% 24 Validation loss: 0.067343 Best loss: 0.048011 Accuracy: 98.51% 25 Validation loss: 0.075505 Best loss: 0.048011 Accuracy: 98.20% 26 Validation loss: 0.068811 Best loss: 0.048011 Accuracy: 98.55% 27 Validation loss: 0.075530 Best loss: 0.048011 Accuracy: 98.59% 28 Validation loss: 0.069468 Best loss: 0.048011 Accuracy: 98.51% 29 Validation loss: 0.095327 Best loss: 0.048011 Accuracy: 98.24% 30 Validation loss: 0.074630 Best loss: 0.048011 Accuracy: 98.32% 31 Validation loss: 0.065945 Best loss: 0.048011 Accuracy: 98.36% 32 Validation loss: 0.059968 Best loss: 0.048011 Accuracy: 98.75% 33 Validation loss: 0.063266 Best loss: 0.048011 Accuracy: 98.83% 34 Validation loss: 0.074900 Best loss: 0.048011 Accuracy: 98.40% 35 Validation loss: 0.072815 Best loss: 0.048011 Accuracy: 98.59% 36 Validation loss: 0.063964 Best loss: 0.048011 Accuracy: 98.71% 37 Validation loss: 0.067646 Best loss: 0.048011 Accuracy: 98.71% 38 Validation loss: 0.070395 Best loss: 0.048011 Accuracy: 98.63% 39 Validation loss: 0.073870 Best loss: 0.048011 Accuracy: 98.55% 40 Validation loss: 0.064028 Best loss: 0.048011 Accuracy: 98.59% 41 Validation loss: 0.079293 Best loss: 0.048011 Accuracy: 98.48% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 13.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.105731 Best loss: 0.105731 Accuracy: 96.40% 1 Validation loss: 0.075197 Best loss: 0.075197 Accuracy: 97.50% 2 Validation loss: 0.070231 Best loss: 0.070231 Accuracy: 97.69% 3 Validation loss: 0.071496 Best loss: 0.070231 Accuracy: 97.89% 4 Validation loss: 0.061722 Best loss: 0.061722 Accuracy: 97.97% 5 Validation loss: 0.065675 Best loss: 0.061722 Accuracy: 97.89% 6 Validation loss: 0.052589 Best loss: 0.052589 Accuracy: 98.44% 7 Validation loss: 0.054083 Best loss: 0.052589 Accuracy: 98.55% 8 Validation loss: 0.055896 Best loss: 0.052589 Accuracy: 98.36% 9 Validation loss: 0.066982 Best loss: 0.052589 Accuracy: 98.36% 10 Validation loss: 0.070403 Best loss: 0.052589 Accuracy: 98.32% 11 Validation loss: 0.082869 Best loss: 0.052589 Accuracy: 98.16% 12 Validation loss: 0.058251 Best loss: 0.052589 Accuracy: 98.63% 13 Validation loss: 0.058958 Best loss: 0.052589 Accuracy: 98.59% 14 Validation loss: 0.068304 Best loss: 0.052589 Accuracy: 98.20% 15 Validation loss: 0.066807 Best loss: 0.052589 Accuracy: 98.55% 16 Validation loss: 0.064942 Best loss: 0.052589 Accuracy: 98.51% 17 Validation loss: 0.087649 Best loss: 0.052589 Accuracy: 98.36% 18 Validation loss: 0.072415 Best loss: 0.052589 Accuracy: 98.16% 19 Validation loss: 0.077595 Best loss: 0.052589 Accuracy: 98.28% 20 Validation loss: 0.078020 Best loss: 0.052589 Accuracy: 98.51% 21 Validation loss: 0.072307 Best loss: 0.052589 Accuracy: 98.55% 22 Validation loss: 0.086823 Best loss: 0.052589 Accuracy: 98.44% 23 Validation loss: 0.060417 Best loss: 0.052589 Accuracy: 98.51% 24 Validation loss: 0.071218 Best loss: 0.052589 Accuracy: 98.55% 25 Validation loss: 0.069028 Best loss: 0.052589 Accuracy: 98.59% 26 Validation loss: 0.067846 Best loss: 0.052589 Accuracy: 98.75% 27 Validation loss: 0.072708 Best loss: 0.052589 Accuracy: 98.51% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 10.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.117643 Best loss: 0.117643 Accuracy: 96.13% 1 Validation loss: 0.084914 Best loss: 0.084914 Accuracy: 97.46% 2 Validation loss: 0.074795 Best loss: 0.074795 Accuracy: 97.58% 3 Validation loss: 0.080473 Best loss: 0.074795 Accuracy: 97.38% 4 Validation loss: 0.055690 Best loss: 0.055690 Accuracy: 98.20% 5 Validation loss: 0.079771 Best loss: 0.055690 Accuracy: 97.38% 6 Validation loss: 0.047808 Best loss: 0.047808 Accuracy: 98.12% 7 Validation loss: 0.044312 Best loss: 0.044312 Accuracy: 98.40% 8 Validation loss: 0.068742 Best loss: 0.044312 Accuracy: 98.16% 9 Validation loss: 0.037305 Best loss: 0.037305 Accuracy: 98.75% 10 Validation loss: 0.055379 Best loss: 0.037305 Accuracy: 98.48% 11 Validation loss: 0.063224 Best loss: 0.037305 Accuracy: 98.44% 12 Validation loss: 0.052222 Best loss: 0.037305 Accuracy: 98.59% 13 Validation loss: 0.071269 Best loss: 0.037305 Accuracy: 98.32% 14 Validation loss: 0.047992 Best loss: 0.037305 Accuracy: 98.67% 15 Validation loss: 0.073533 Best loss: 0.037305 Accuracy: 98.36% 16 Validation loss: 0.061717 Best loss: 0.037305 Accuracy: 98.63% 17 Validation loss: 0.072180 Best loss: 0.037305 Accuracy: 98.55% 18 Validation loss: 0.069867 Best loss: 0.037305 Accuracy: 98.55% 19 Validation loss: 0.071543 Best loss: 0.037305 Accuracy: 98.24% 20 Validation loss: 0.069336 Best loss: 0.037305 Accuracy: 98.48% 21 Validation loss: 0.061683 Best loss: 0.037305 Accuracy: 98.51% 22 Validation loss: 0.083573 Best loss: 0.037305 Accuracy: 98.32% 23 Validation loss: 0.065702 Best loss: 0.037305 Accuracy: 98.51% 24 Validation loss: 0.067678 Best loss: 0.037305 Accuracy: 98.40% 25 Validation loss: 0.064549 Best loss: 0.037305 Accuracy: 98.71% 26 Validation loss: 0.066077 Best loss: 0.037305 Accuracy: 98.75% 27 Validation loss: 0.061140 Best loss: 0.037305 Accuracy: 98.79% 28 Validation loss: 0.077822 Best loss: 0.037305 Accuracy: 98.48% 29 Validation loss: 0.080986 Best loss: 0.037305 Accuracy: 98.40% 30 Validation loss: 0.060646 Best loss: 0.037305 Accuracy: 98.59% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 11.4s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.091406 Best loss: 0.091406 Accuracy: 97.11% 1 Validation loss: 0.067091 Best loss: 0.067091 Accuracy: 98.01% 2 Validation loss: 0.058965 Best loss: 0.058965 Accuracy: 98.08% 3 Validation loss: 0.045672 Best loss: 0.045672 Accuracy: 98.44% 4 Validation loss: 0.075257 Best loss: 0.045672 Accuracy: 97.58% 5 Validation loss: 0.048883 Best loss: 0.045672 Accuracy: 98.51% 6 Validation loss: 0.054660 Best loss: 0.045672 Accuracy: 98.55% 7 Validation loss: 0.060687 Best loss: 0.045672 Accuracy: 98.44% 8 Validation loss: 0.072963 Best loss: 0.045672 Accuracy: 98.05% 9 Validation loss: 0.053601 Best loss: 0.045672 Accuracy: 98.51% 10 Validation loss: 0.064435 Best loss: 0.045672 Accuracy: 98.48% 11 Validation loss: 0.043901 Best loss: 0.043901 Accuracy: 98.83% 12 Validation loss: 0.064602 Best loss: 0.043901 Accuracy: 98.40% 13 Validation loss: 0.060959 Best loss: 0.043901 Accuracy: 98.24% 14 Validation loss: 0.060515 Best loss: 0.043901 Accuracy: 98.55% 15 Validation loss: 0.060820 Best loss: 0.043901 Accuracy: 98.40% 16 Validation loss: 0.054976 Best loss: 0.043901 Accuracy: 98.79% 17 Validation loss: 0.058113 Best loss: 0.043901 Accuracy: 98.67% 18 Validation loss: 0.042667 Best loss: 0.042667 Accuracy: 98.87% 19 Validation loss: 0.064954 Best loss: 0.042667 Accuracy: 98.36% 20 Validation loss: 0.046482 Best loss: 0.042667 Accuracy: 99.14% 21 Validation loss: 0.061186 Best loss: 0.042667 Accuracy: 98.87% 22 Validation loss: 0.035600 Best loss: 0.035600 Accuracy: 99.10% 23 Validation loss: 0.049631 Best loss: 0.035600 Accuracy: 99.10% 24 Validation loss: 0.055339 Best loss: 0.035600 Accuracy: 98.67% 25 Validation loss: 0.058940 Best loss: 0.035600 Accuracy: 98.67% 26 Validation loss: 0.058570 Best loss: 0.035600 Accuracy: 98.71% 27 Validation loss: 0.051610 Best loss: 0.035600 Accuracy: 98.94% 28 Validation loss: 0.050124 Best loss: 0.035600 Accuracy: 98.71% 29 Validation loss: 0.060603 Best loss: 0.035600 Accuracy: 98.79% 30 Validation loss: 0.049852 Best loss: 0.035600 Accuracy: 98.87% 31 Validation loss: 0.056368 Best loss: 0.035600 Accuracy: 98.98% 32 Validation loss: 0.063122 Best loss: 0.035600 Accuracy: 98.87% 33 Validation loss: 0.067719 Best loss: 0.035600 Accuracy: 98.87% 34 Validation loss: 0.086255 Best loss: 0.035600 Accuracy: 98.36% 35 Validation loss: 0.064056 Best loss: 0.035600 Accuracy: 98.87% 36 Validation loss: 0.068112 Best loss: 0.035600 Accuracy: 98.91% 37 Validation loss: 0.063344 Best loss: 0.035600 Accuracy: 98.71% 38 Validation loss: 0.069582 Best loss: 0.035600 Accuracy: 98.83% 39 Validation loss: 0.042205 Best loss: 0.035600 Accuracy: 99.14% 40 Validation loss: 0.049414 Best loss: 0.035600 Accuracy: 98.94% 41 Validation loss: 0.084555 Best loss: 0.035600 Accuracy: 98.83% 42 Validation loss: 0.050083 Best loss: 0.035600 Accuracy: 99.14% 43 Validation loss: 0.050076 Best loss: 0.035600 Accuracy: 98.79% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 51.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.181272 Best loss: 0.181272 Accuracy: 95.35% 1 Validation loss: 0.062951 Best loss: 0.062951 Accuracy: 98.24% 2 Validation loss: 0.060392 Best loss: 0.060392 Accuracy: 98.24% 3 Validation loss: 0.078027 Best loss: 0.060392 Accuracy: 97.69% 4 Validation loss: 0.092143 Best loss: 0.060392 Accuracy: 97.11% 5 Validation loss: 0.043753 Best loss: 0.043753 Accuracy: 98.63% 6 Validation loss: 0.057600 Best loss: 0.043753 Accuracy: 98.44% 7 Validation loss: 0.090058 Best loss: 0.043753 Accuracy: 97.73% 8 Validation loss: 0.072071 Best loss: 0.043753 Accuracy: 98.28% 9 Validation loss: 0.107379 Best loss: 0.043753 Accuracy: 97.50% 10 Validation loss: 0.042561 Best loss: 0.042561 Accuracy: 98.91% 11 Validation loss: 0.050341 Best loss: 0.042561 Accuracy: 98.63% 12 Validation loss: 0.048675 Best loss: 0.042561 Accuracy: 98.87% 13 Validation loss: 0.066631 Best loss: 0.042561 Accuracy: 98.40% 14 Validation loss: 0.045548 Best loss: 0.042561 Accuracy: 98.83% 15 Validation loss: 0.065467 Best loss: 0.042561 Accuracy: 98.51% 16 Validation loss: 0.068101 Best loss: 0.042561 Accuracy: 98.51% 17 Validation loss: 0.052514 Best loss: 0.042561 Accuracy: 98.94% 18 Validation loss: 0.112505 Best loss: 0.042561 Accuracy: 97.58% 19 Validation loss: 0.039501 Best loss: 0.039501 Accuracy: 98.94% 20 Validation loss: 0.055958 Best loss: 0.039501 Accuracy: 98.71% 21 Validation loss: 0.046060 Best loss: 0.039501 Accuracy: 98.98% 22 Validation loss: 0.058631 Best loss: 0.039501 Accuracy: 98.75% 23 Validation loss: 0.049388 Best loss: 0.039501 Accuracy: 98.94% 24 Validation loss: 0.061389 Best loss: 0.039501 Accuracy: 98.63% 25 Validation loss: 0.039378 Best loss: 0.039378 Accuracy: 98.98% 26 Validation loss: 0.054401 Best loss: 0.039378 Accuracy: 98.87% 27 Validation loss: 0.054123 Best loss: 0.039378 Accuracy: 98.94% 28 Validation loss: 0.051443 Best loss: 0.039378 Accuracy: 98.75% 29 Validation loss: 0.070915 Best loss: 0.039378 Accuracy: 98.63% 30 Validation loss: 0.052864 Best loss: 0.039378 Accuracy: 98.98% 31 Validation loss: 0.057502 Best loss: 0.039378 Accuracy: 98.87% 32 Validation loss: 0.049876 Best loss: 0.039378 Accuracy: 99.02% 33 Validation loss: 0.046606 Best loss: 0.039378 Accuracy: 99.02% 34 Validation loss: 0.034274 Best loss: 0.034274 Accuracy: 99.26% 35 Validation loss: 0.051781 Best loss: 0.034274 Accuracy: 98.94% 36 Validation loss: 0.068427 Best loss: 0.034274 Accuracy: 98.44% 37 Validation loss: 0.052282 Best loss: 0.034274 Accuracy: 99.02% 38 Validation loss: 0.051231 Best loss: 0.034274 Accuracy: 99.06% 39 Validation loss: 0.052919 Best loss: 0.034274 Accuracy: 98.98% 40 Validation loss: 0.054280 Best loss: 0.034274 Accuracy: 98.98% 41 Validation loss: 0.044287 Best loss: 0.034274 Accuracy: 99.22% 42 Validation loss: 0.052211 Best loss: 0.034274 Accuracy: 98.98% 43 Validation loss: 0.077858 Best loss: 0.034274 Accuracy: 98.59% 44 Validation loss: 0.056661 Best loss: 0.034274 Accuracy: 98.94% 45 Validation loss: 0.066231 Best loss: 0.034274 Accuracy: 98.75% 46 Validation loss: 0.063144 Best loss: 0.034274 Accuracy: 98.83% 47 Validation loss: 0.057991 Best loss: 0.034274 Accuracy: 99.14% 48 Validation loss: 0.050893 Best loss: 0.034274 Accuracy: 99.14% 49 Validation loss: 0.056824 Best loss: 0.034274 Accuracy: 98.87% 50 Validation loss: 0.048521 Best loss: 0.034274 Accuracy: 99.14% 51 Validation loss: 0.055010 Best loss: 0.034274 Accuracy: 99.22% 52 Validation loss: 0.062189 Best loss: 0.034274 Accuracy: 98.94% 53 Validation loss: 0.055460 Best loss: 0.034274 Accuracy: 99.18% 54 Validation loss: 0.069965 Best loss: 0.034274 Accuracy: 98.87% 55 Validation loss: 0.086624 Best loss: 0.034274 Accuracy: 97.93% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 1.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02 0 Validation loss: 0.097698 Best loss: 0.097698 Accuracy: 97.07% 1 Validation loss: 0.082933 Best loss: 0.082933 Accuracy: 97.62% 2 Validation loss: 0.065622 Best loss: 0.065622 Accuracy: 97.81% 3 Validation loss: 0.057923 Best loss: 0.057923 Accuracy: 98.24% 4 Validation loss: 0.052170 Best loss: 0.052170 Accuracy: 98.44% 5 Validation loss: 0.043012 Best loss: 0.043012 Accuracy: 98.59% 6 Validation loss: 0.042059 Best loss: 0.042059 Accuracy: 98.87% 7 Validation loss: 0.070427 Best loss: 0.042059 Accuracy: 98.12% 8 Validation loss: 0.054740 Best loss: 0.042059 Accuracy: 98.71% 9 Validation loss: 0.047806 Best loss: 0.042059 Accuracy: 98.48% 10 Validation loss: 0.056015 Best loss: 0.042059 Accuracy: 98.48% 11 Validation loss: 0.046797 Best loss: 0.042059 Accuracy: 98.79% 12 Validation loss: 0.061353 Best loss: 0.042059 Accuracy: 98.75% 13 Validation loss: 0.052215 Best loss: 0.042059 Accuracy: 98.91% 14 Validation loss: 0.041912 Best loss: 0.041912 Accuracy: 98.98% 15 Validation loss: 0.060233 Best loss: 0.041912 Accuracy: 98.75% 16 Validation loss: 0.051820 Best loss: 0.041912 Accuracy: 98.94% 17 Validation loss: 0.042098 Best loss: 0.041912 Accuracy: 98.87% 18 Validation loss: 0.045613 Best loss: 0.041912 Accuracy: 98.98% 19 Validation loss: 0.049907 Best loss: 0.041912 Accuracy: 98.87% 20 Validation loss: 0.086688 Best loss: 0.041912 Accuracy: 98.24% 21 Validation loss: 0.069166 Best loss: 0.041912 Accuracy: 98.40% 22 Validation loss: 0.041864 Best loss: 0.041864 Accuracy: 99.02% 23 Validation loss: 0.074650 Best loss: 0.041864 Accuracy: 98.48% 24 Validation loss: 0.052877 Best loss: 0.041864 Accuracy: 98.98% 25 Validation loss: 0.052241 Best loss: 0.041864 Accuracy: 98.63% 26 Validation loss: 0.043025 Best loss: 0.041864 Accuracy: 99.10% 27 Validation loss: 0.067076 Best loss: 0.041864 Accuracy: 98.67% 28 Validation loss: 0.099974 Best loss: 0.041864 Accuracy: 98.59% 29 Validation loss: 0.042954 Best loss: 0.041864 Accuracy: 99.02% 30 Validation loss: 0.034889 Best loss: 0.034889 Accuracy: 98.94% 31 Validation loss: 0.040641 Best loss: 0.034889 Accuracy: 99.22% 32 Validation loss: 0.053143 Best loss: 0.034889 Accuracy: 98.94% 33 Validation loss: 0.049958 Best loss: 0.034889 Accuracy: 99.14% 34 Validation loss: 0.046541 Best loss: 0.034889 Accuracy: 99.37% 35 Validation loss: 0.058078 Best loss: 0.034889 Accuracy: 99.10% 36 Validation loss: 0.054457 Best loss: 0.034889 Accuracy: 99.02% 37 Validation loss: 0.046135 Best loss: 0.034889 Accuracy: 99.30% 38 Validation loss: 0.052015 Best loss: 0.034889 Accuracy: 98.79% 39 Validation loss: 0.052489 Best loss: 0.034889 Accuracy: 99.10% 40 Validation loss: 0.055142 Best loss: 0.034889 Accuracy: 99.02% 41 Validation loss: 0.061958 Best loss: 0.034889 Accuracy: 98.98% 42 Validation loss: 0.049211 Best loss: 0.034889 Accuracy: 99.02% 43 Validation loss: 0.029203 Best loss: 0.029203 Accuracy: 99.34% 44 Validation loss: 0.034747 Best loss: 0.029203 Accuracy: 99.53% 45 Validation loss: 0.046784 Best loss: 0.029203 Accuracy: 98.91% 46 Validation loss: 0.038704 Best loss: 0.029203 Accuracy: 99.22% 47 Validation loss: 0.028260 Best loss: 0.028260 Accuracy: 99.45% 48 Validation loss: 0.046551 Best loss: 0.028260 Accuracy: 99.06% 49 Validation loss: 0.034224 Best loss: 0.028260 Accuracy: 99.26% 50 Validation loss: 0.049192 Best loss: 0.028260 Accuracy: 99.10% 51 Validation loss: 0.044094 Best loss: 0.028260 Accuracy: 99.18% 52 Validation loss: 0.035963 Best loss: 0.028260 Accuracy: 99.14% 53 Validation loss: 0.033736 Best loss: 0.028260 Accuracy: 99.34% 54 Validation loss: 0.070991 Best loss: 0.028260 Accuracy: 98.87% 55 Validation loss: 0.058462 Best loss: 0.028260 Accuracy: 99.06% 56 Validation loss: 0.052606 Best loss: 0.028260 Accuracy: 99.06% 57 Validation loss: 0.049868 Best loss: 0.028260 Accuracy: 99.14% 58 Validation loss: 0.046381 Best loss: 0.028260 Accuracy: 99.26% 59 Validation loss: 0.045297 Best loss: 0.028260 Accuracy: 99.26% 60 Validation loss: 0.045194 Best loss: 0.028260 Accuracy: 99.22% 61 Validation loss: 0.048286 Best loss: 0.028260 Accuracy: 99.02% 62 Validation loss: 0.055049 Best loss: 0.028260 Accuracy: 99.06% 63 Validation loss: 0.052622 Best loss: 0.028260 Accuracy: 99.10% 64 Validation loss: 0.085472 Best loss: 0.028260 Accuracy: 98.63% 65 Validation loss: 0.085128 Best loss: 0.028260 Accuracy: 98.83% 66 Validation loss: 0.057077 Best loss: 0.028260 Accuracy: 98.91% 67 Validation loss: 0.049205 Best loss: 0.028260 Accuracy: 99.10% 68 Validation loss: 0.045377 Best loss: 0.028260 Accuracy: 99.14% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, batch_norm_momentum=0.95, learning_rate=0.02, total= 1.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.132292 Best loss: 0.132292 Accuracy: 96.21% 1 Validation loss: 0.086607 Best loss: 0.086607 Accuracy: 97.50% 2 Validation loss: 0.126429 Best loss: 0.086607 Accuracy: 97.58% 3 Validation loss: 0.157849 Best loss: 0.086607 Accuracy: 96.52% 4 Validation loss: 0.113189 Best loss: 0.086607 Accuracy: 98.01% 5 Validation loss: 0.129370 Best loss: 0.086607 Accuracy: 97.30% 6 Validation loss: 0.674579 Best loss: 0.086607 Accuracy: 93.82% 7 Validation loss: 0.261778 Best loss: 0.086607 Accuracy: 98.20% 8 Validation loss: 0.065834 Best loss: 0.065834 Accuracy: 98.48% 9 Validation loss: 0.058838 Best loss: 0.058838 Accuracy: 98.87% 10 Validation loss: 0.046809 Best loss: 0.046809 Accuracy: 98.91% 11 Validation loss: 0.050304 Best loss: 0.046809 Accuracy: 98.87% 12 Validation loss: 0.071870 Best loss: 0.046809 Accuracy: 98.51% 13 Validation loss: 0.103216 Best loss: 0.046809 Accuracy: 98.59% 14 Validation loss: 7.663714 Best loss: 0.046809 Accuracy: 84.87% 15 Validation loss: 0.227113 Best loss: 0.046809 Accuracy: 98.01% 16 Validation loss: 0.118511 Best loss: 0.046809 Accuracy: 98.51% 17 Validation loss: 0.101107 Best loss: 0.046809 Accuracy: 98.87% 18 Validation loss: 0.089316 Best loss: 0.046809 Accuracy: 98.79% 19 Validation loss: 0.072827 Best loss: 0.046809 Accuracy: 98.94% 20 Validation loss: 0.077891 Best loss: 0.046809 Accuracy: 98.75% 21 Validation loss: 0.079807 Best loss: 0.046809 Accuracy: 98.67% 22 Validation loss: 0.100235 Best loss: 0.046809 Accuracy: 98.67% 23 Validation loss: 1.107645 Best loss: 0.046809 Accuracy: 96.64% 24 Validation loss: 0.206610 Best loss: 0.046809 Accuracy: 98.40% 25 Validation loss: 0.151243 Best loss: 0.046809 Accuracy: 98.67% 26 Validation loss: 0.094409 Best loss: 0.046809 Accuracy: 98.83% 27 Validation loss: 0.205374 Best loss: 0.046809 Accuracy: 98.28% 28 Validation loss: 0.281866 Best loss: 0.046809 Accuracy: 98.05% 29 Validation loss: 0.741537 Best loss: 0.046809 Accuracy: 98.40% 30 Validation loss: 0.314582 Best loss: 0.046809 Accuracy: 98.48% 31 Validation loss: 0.163063 Best loss: 0.046809 Accuracy: 98.55% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.101678 Best loss: 0.101678 Accuracy: 97.11% 1 Validation loss: 0.079762 Best loss: 0.079762 Accuracy: 97.97% 2 Validation loss: 0.100833 Best loss: 0.079762 Accuracy: 97.97% 3 Validation loss: 0.066104 Best loss: 0.066104 Accuracy: 97.93% 4 Validation loss: 0.271530 Best loss: 0.066104 Accuracy: 95.90% 5 Validation loss: 0.228246 Best loss: 0.066104 Accuracy: 97.85% 6 Validation loss: 0.108424 Best loss: 0.066104 Accuracy: 98.51% 7 Validation loss: 0.041966 Best loss: 0.041966 Accuracy: 98.75% 8 Validation loss: 0.047549 Best loss: 0.041966 Accuracy: 98.94% 9 Validation loss: 0.047670 Best loss: 0.041966 Accuracy: 99.06% 10 Validation loss: 0.050264 Best loss: 0.041966 Accuracy: 98.83% 11 Validation loss: 0.059335 Best loss: 0.041966 Accuracy: 98.67% 12 Validation loss: 0.100038 Best loss: 0.041966 Accuracy: 98.51% 13 Validation loss: 2.025001 Best loss: 0.041966 Accuracy: 96.87% 14 Validation loss: 0.128252 Best loss: 0.041966 Accuracy: 98.75% 15 Validation loss: 0.070703 Best loss: 0.041966 Accuracy: 98.83% 16 Validation loss: 0.057128 Best loss: 0.041966 Accuracy: 99.06% 17 Validation loss: 0.069441 Best loss: 0.041966 Accuracy: 98.79% 18 Validation loss: 0.087689 Best loss: 0.041966 Accuracy: 98.67% 19 Validation loss: 0.061452 Best loss: 0.041966 Accuracy: 98.98% 20 Validation loss: 0.063093 Best loss: 0.041966 Accuracy: 98.91% 21 Validation loss: 0.092849 Best loss: 0.041966 Accuracy: 98.75% 22 Validation loss: 21.132668 Best loss: 0.041966 Accuracy: 91.79% 23 Validation loss: 0.465763 Best loss: 0.041966 Accuracy: 98.32% 24 Validation loss: 0.352925 Best loss: 0.041966 Accuracy: 98.24% 25 Validation loss: 0.153026 Best loss: 0.041966 Accuracy: 98.87% 26 Validation loss: 0.263588 Best loss: 0.041966 Accuracy: 98.55% 27 Validation loss: 0.186905 Best loss: 0.041966 Accuracy: 98.87% 28 Validation loss: 0.181320 Best loss: 0.041966 Accuracy: 98.51% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.1min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.239966 Best loss: 0.239966 Accuracy: 94.84% 1 Validation loss: 0.131873 Best loss: 0.131873 Accuracy: 96.36% 2 Validation loss: 0.070582 Best loss: 0.070582 Accuracy: 98.08% 3 Validation loss: 0.073982 Best loss: 0.070582 Accuracy: 98.05% 4 Validation loss: 0.427240 Best loss: 0.070582 Accuracy: 96.52% 5 Validation loss: 0.075372 Best loss: 0.070582 Accuracy: 98.16% 6 Validation loss: 0.151967 Best loss: 0.070582 Accuracy: 96.64% 7 Validation loss: 0.057937 Best loss: 0.057937 Accuracy: 98.71% 8 Validation loss: 0.077096 Best loss: 0.057937 Accuracy: 98.48% 9 Validation loss: 0.407080 Best loss: 0.057937 Accuracy: 97.97% 10 Validation loss: 0.305529 Best loss: 0.057937 Accuracy: 98.08% 11 Validation loss: 0.051879 Best loss: 0.051879 Accuracy: 98.59% 12 Validation loss: 0.036232 Best loss: 0.036232 Accuracy: 99.18% 13 Validation loss: 0.038715 Best loss: 0.036232 Accuracy: 99.14% 14 Validation loss: 0.082176 Best loss: 0.036232 Accuracy: 98.67% 15 Validation loss: 41.007263 Best loss: 0.036232 Accuracy: 72.83% 16 Validation loss: 0.140699 Best loss: 0.036232 Accuracy: 98.79% 17 Validation loss: 0.293032 Best loss: 0.036232 Accuracy: 97.81% 18 Validation loss: 0.130264 Best loss: 0.036232 Accuracy: 98.59% 19 Validation loss: 0.154254 Best loss: 0.036232 Accuracy: 99.06% 20 Validation loss: 0.109100 Best loss: 0.036232 Accuracy: 98.91% 21 Validation loss: 0.120280 Best loss: 0.036232 Accuracy: 98.63% 22 Validation loss: 0.172211 Best loss: 0.036232 Accuracy: 98.63% 23 Validation loss: 0.122199 Best loss: 0.036232 Accuracy: 98.79% 24 Validation loss: 0.703587 Best loss: 0.036232 Accuracy: 98.36% 25 Validation loss: 0.168653 Best loss: 0.036232 Accuracy: 98.67% 26 Validation loss: 0.073523 Best loss: 0.036232 Accuracy: 99.37% 27 Validation loss: 0.075605 Best loss: 0.036232 Accuracy: 99.06% 28 Validation loss: 0.281544 Best loss: 0.036232 Accuracy: 98.59% 29 Validation loss: 0.664895 Best loss: 0.036232 Accuracy: 98.16% 30 Validation loss: 0.193009 Best loss: 0.036232 Accuracy: 98.83% 31 Validation loss: 0.169664 Best loss: 0.036232 Accuracy: 98.71% 32 Validation loss: 0.114718 Best loss: 0.036232 Accuracy: 98.91% 33 Validation loss: 0.161067 Best loss: 0.036232 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02 0 Validation loss: 0.215477 Best loss: 0.215477 Accuracy: 93.20% 1 Validation loss: 0.224102 Best loss: 0.215477 Accuracy: 93.39% 2 Validation loss: 0.163887 Best loss: 0.163887 Accuracy: 94.88% 3 Validation loss: 0.101959 Best loss: 0.101959 Accuracy: 97.07% 4 Validation loss: 0.114008 Best loss: 0.101959 Accuracy: 96.72% 5 Validation loss: 0.095905 Best loss: 0.095905 Accuracy: 97.22% 6 Validation loss: 0.085266 Best loss: 0.085266 Accuracy: 98.01% 7 Validation loss: 0.095385 Best loss: 0.085266 Accuracy: 97.54% 8 Validation loss: 0.083061 Best loss: 0.083061 Accuracy: 98.01% 9 Validation loss: 0.144656 Best loss: 0.083061 Accuracy: 96.76% 10 Validation loss: 0.083722 Best loss: 0.083061 Accuracy: 97.81% 11 Validation loss: 0.087396 Best loss: 0.083061 Accuracy: 97.81% 12 Validation loss: 0.122401 Best loss: 0.083061 Accuracy: 96.79% 13 Validation loss: 0.141425 Best loss: 0.083061 Accuracy: 97.07% 14 Validation loss: 0.086270 Best loss: 0.083061 Accuracy: 97.81% 15 Validation loss: 0.089753 Best loss: 0.083061 Accuracy: 98.08% 16 Validation loss: 0.111754 Best loss: 0.083061 Accuracy: 97.58% 17 Validation loss: 0.086158 Best loss: 0.083061 Accuracy: 97.85% 18 Validation loss: 0.092606 Best loss: 0.083061 Accuracy: 97.77% 19 Validation loss: 0.088975 Best loss: 0.083061 Accuracy: 97.89% 20 Validation loss: 0.078330 Best loss: 0.078330 Accuracy: 98.28% 21 Validation loss: 0.080406 Best loss: 0.078330 Accuracy: 98.28% 22 Validation loss: 0.126409 Best loss: 0.078330 Accuracy: 97.58% 23 Validation loss: 0.102883 Best loss: 0.078330 Accuracy: 98.20% 24 Validation loss: 0.096095 Best loss: 0.078330 Accuracy: 97.97% 25 Validation loss: 0.113182 Best loss: 0.078330 Accuracy: 97.69% 26 Validation loss: 0.088670 Best loss: 0.078330 Accuracy: 97.93% 27 Validation loss: 0.100672 Best loss: 0.078330 Accuracy: 98.24% 28 Validation loss: 0.114947 Best loss: 0.078330 Accuracy: 97.89% 29 Validation loss: 0.111563 Best loss: 0.078330 Accuracy: 97.85% 30 Validation loss: 0.101595 Best loss: 0.078330 Accuracy: 97.85% 31 Validation loss: 0.097842 Best loss: 0.078330 Accuracy: 98.12% 32 Validation loss: 0.104097 Best loss: 0.078330 Accuracy: 98.01% 33 Validation loss: 0.098682 Best loss: 0.078330 Accuracy: 98.16% 34 Validation loss: 0.144130 Best loss: 0.078330 Accuracy: 96.79% 35 Validation loss: 0.153634 Best loss: 0.078330 Accuracy: 97.46% 36 Validation loss: 0.118520 Best loss: 0.078330 Accuracy: 98.16% 37 Validation loss: 0.107116 Best loss: 0.078330 Accuracy: 97.73% 38 Validation loss: 0.102812 Best loss: 0.078330 Accuracy: 97.89% 39 Validation loss: 0.101694 Best loss: 0.078330 Accuracy: 98.08% 40 Validation loss: 0.110543 Best loss: 0.078330 Accuracy: 97.93% 41 Validation loss: 0.113938 Best loss: 0.078330 Accuracy: 97.93% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02, total= 13.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02 0 Validation loss: 0.204026 Best loss: 0.204026 Accuracy: 93.78% 1 Validation loss: 0.113345 Best loss: 0.113345 Accuracy: 96.72% 2 Validation loss: 0.118682 Best loss: 0.113345 Accuracy: 95.97% 3 Validation loss: 0.244170 Best loss: 0.113345 Accuracy: 91.28% 4 Validation loss: 0.122097 Best loss: 0.113345 Accuracy: 96.33% 5 Validation loss: 0.103995 Best loss: 0.103995 Accuracy: 97.07% 6 Validation loss: 0.138487 Best loss: 0.103995 Accuracy: 96.13% 7 Validation loss: 0.141927 Best loss: 0.103995 Accuracy: 95.90% 8 Validation loss: 0.216650 Best loss: 0.103995 Accuracy: 94.02% 9 Validation loss: 0.137929 Best loss: 0.103995 Accuracy: 96.29% 10 Validation loss: 0.099232 Best loss: 0.099232 Accuracy: 97.73% 11 Validation loss: 0.160520 Best loss: 0.099232 Accuracy: 96.09% 12 Validation loss: 0.146755 Best loss: 0.099232 Accuracy: 96.56% 13 Validation loss: 0.188408 Best loss: 0.099232 Accuracy: 95.82% 14 Validation loss: 0.134052 Best loss: 0.099232 Accuracy: 96.56% 15 Validation loss: 0.159919 Best loss: 0.099232 Accuracy: 96.29% 16 Validation loss: 0.099501 Best loss: 0.099232 Accuracy: 97.65% 17 Validation loss: 0.105859 Best loss: 0.099232 Accuracy: 97.58% 18 Validation loss: 0.086250 Best loss: 0.086250 Accuracy: 97.85% 19 Validation loss: 0.112672 Best loss: 0.086250 Accuracy: 97.81% 20 Validation loss: 0.095610 Best loss: 0.086250 Accuracy: 97.85% 21 Validation loss: 0.166008 Best loss: 0.086250 Accuracy: 96.52% 22 Validation loss: 0.121467 Best loss: 0.086250 Accuracy: 97.69% 23 Validation loss: 0.120894 Best loss: 0.086250 Accuracy: 97.73% 24 Validation loss: 0.097220 Best loss: 0.086250 Accuracy: 97.85% 25 Validation loss: 0.103467 Best loss: 0.086250 Accuracy: 97.69% 26 Validation loss: 0.155122 Best loss: 0.086250 Accuracy: 97.07% 27 Validation loss: 0.110113 Best loss: 0.086250 Accuracy: 97.26% 28 Validation loss: 0.095511 Best loss: 0.086250 Accuracy: 97.73% 29 Validation loss: 0.106416 Best loss: 0.086250 Accuracy: 97.50% 30 Validation loss: 0.122156 Best loss: 0.086250 Accuracy: 97.58% 31 Validation loss: 0.114044 Best loss: 0.086250 Accuracy: 97.62% 32 Validation loss: 0.110508 Best loss: 0.086250 Accuracy: 97.73% 33 Validation loss: 0.137245 Best loss: 0.086250 Accuracy: 97.07% 34 Validation loss: 0.090876 Best loss: 0.086250 Accuracy: 98.08% 35 Validation loss: 0.132010 Best loss: 0.086250 Accuracy: 97.22% 36 Validation loss: 0.119975 Best loss: 0.086250 Accuracy: 97.77% 37 Validation loss: 0.153436 Best loss: 0.086250 Accuracy: 97.19% 38 Validation loss: 0.111741 Best loss: 0.086250 Accuracy: 97.54% 39 Validation loss: 0.111863 Best loss: 0.086250 Accuracy: 97.58% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02, total= 14.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02 0 Validation loss: 0.190293 Best loss: 0.190293 Accuracy: 93.90% 1 Validation loss: 0.115565 Best loss: 0.115565 Accuracy: 96.76% 2 Validation loss: 0.168829 Best loss: 0.115565 Accuracy: 94.88% 3 Validation loss: 0.136131 Best loss: 0.115565 Accuracy: 95.82% 4 Validation loss: 0.136485 Best loss: 0.115565 Accuracy: 96.09% 5 Validation loss: 0.117994 Best loss: 0.115565 Accuracy: 96.56% 6 Validation loss: 0.186258 Best loss: 0.115565 Accuracy: 94.80% 7 Validation loss: 0.103862 Best loss: 0.103862 Accuracy: 97.11% 8 Validation loss: 0.122098 Best loss: 0.103862 Accuracy: 96.83% 9 Validation loss: 0.094125 Best loss: 0.094125 Accuracy: 97.62% 10 Validation loss: 0.087964 Best loss: 0.087964 Accuracy: 97.38% 11 Validation loss: 0.101030 Best loss: 0.087964 Accuracy: 97.26% 12 Validation loss: 0.104969 Best loss: 0.087964 Accuracy: 97.42% 13 Validation loss: 0.126307 Best loss: 0.087964 Accuracy: 97.07% 14 Validation loss: 0.126824 Best loss: 0.087964 Accuracy: 97.26% 15 Validation loss: 0.083402 Best loss: 0.083402 Accuracy: 98.12% 16 Validation loss: 0.091715 Best loss: 0.083402 Accuracy: 97.85% 17 Validation loss: 0.079865 Best loss: 0.079865 Accuracy: 97.85% 18 Validation loss: 0.106439 Best loss: 0.079865 Accuracy: 97.65% 19 Validation loss: 0.115526 Best loss: 0.079865 Accuracy: 97.38% 20 Validation loss: 0.102158 Best loss: 0.079865 Accuracy: 97.77% 21 Validation loss: 0.095398 Best loss: 0.079865 Accuracy: 97.85% 22 Validation loss: 0.122224 Best loss: 0.079865 Accuracy: 97.54% 23 Validation loss: 0.095579 Best loss: 0.079865 Accuracy: 98.05% 24 Validation loss: 0.102850 Best loss: 0.079865 Accuracy: 97.62% 25 Validation loss: 0.101335 Best loss: 0.079865 Accuracy: 97.97% 26 Validation loss: 0.110186 Best loss: 0.079865 Accuracy: 98.08% 27 Validation loss: 0.113932 Best loss: 0.079865 Accuracy: 97.69% 28 Validation loss: 0.102660 Best loss: 0.079865 Accuracy: 98.05% 29 Validation loss: 0.095948 Best loss: 0.079865 Accuracy: 98.12% 30 Validation loss: 0.096484 Best loss: 0.079865 Accuracy: 98.08% 31 Validation loss: 0.108284 Best loss: 0.079865 Accuracy: 97.89% 32 Validation loss: 0.094896 Best loss: 0.079865 Accuracy: 98.08% 33 Validation loss: 0.080037 Best loss: 0.079865 Accuracy: 98.40% 34 Validation loss: 0.123943 Best loss: 0.079865 Accuracy: 97.58% 35 Validation loss: 0.097530 Best loss: 0.079865 Accuracy: 97.89% 36 Validation loss: 0.122761 Best loss: 0.079865 Accuracy: 97.54% 37 Validation loss: 0.109881 Best loss: 0.079865 Accuracy: 98.05% 38 Validation loss: 0.115303 Best loss: 0.079865 Accuracy: 97.89% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.02, total= 13.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.128583 Best loss: 0.128583 Accuracy: 96.79% 1 Validation loss: 0.066592 Best loss: 0.066592 Accuracy: 98.28% 2 Validation loss: 0.086699 Best loss: 0.066592 Accuracy: 97.69% 3 Validation loss: 0.081327 Best loss: 0.066592 Accuracy: 97.69% 4 Validation loss: 0.068821 Best loss: 0.066592 Accuracy: 98.08% 5 Validation loss: 0.065314 Best loss: 0.065314 Accuracy: 98.32% 6 Validation loss: 0.053422 Best loss: 0.053422 Accuracy: 98.55% 7 Validation loss: 0.101106 Best loss: 0.053422 Accuracy: 97.58% 8 Validation loss: 0.218766 Best loss: 0.053422 Accuracy: 97.34% 9 Validation loss: 0.072954 Best loss: 0.053422 Accuracy: 98.67% 10 Validation loss: 0.070756 Best loss: 0.053422 Accuracy: 98.51% 11 Validation loss: 0.066788 Best loss: 0.053422 Accuracy: 98.24% 12 Validation loss: 0.100715 Best loss: 0.053422 Accuracy: 98.28% 13 Validation loss: 0.066038 Best loss: 0.053422 Accuracy: 98.48% 14 Validation loss: 0.123005 Best loss: 0.053422 Accuracy: 97.93% 15 Validation loss: 0.067679 Best loss: 0.053422 Accuracy: 98.79% 16 Validation loss: 0.042150 Best loss: 0.042150 Accuracy: 99.10% 17 Validation loss: 0.065253 Best loss: 0.042150 Accuracy: 98.75% 18 Validation loss: 0.133134 Best loss: 0.042150 Accuracy: 98.32% 19 Validation loss: 0.059920 Best loss: 0.042150 Accuracy: 98.79% 20 Validation loss: 0.064803 Best loss: 0.042150 Accuracy: 98.83% 21 Validation loss: 0.054766 Best loss: 0.042150 Accuracy: 98.87% 22 Validation loss: 0.068745 Best loss: 0.042150 Accuracy: 98.67% 23 Validation loss: 0.179965 Best loss: 0.042150 Accuracy: 98.79% 24 Validation loss: 0.057615 Best loss: 0.042150 Accuracy: 98.98% 25 Validation loss: 0.084344 Best loss: 0.042150 Accuracy: 98.79% 26 Validation loss: 0.077938 Best loss: 0.042150 Accuracy: 98.91% 27 Validation loss: 0.094373 Best loss: 0.042150 Accuracy: 99.02% 28 Validation loss: 0.126754 Best loss: 0.042150 Accuracy: 98.63% 29 Validation loss: 0.139398 Best loss: 0.042150 Accuracy: 98.44% 30 Validation loss: 0.073015 Best loss: 0.042150 Accuracy: 98.94% 31 Validation loss: 0.094123 Best loss: 0.042150 Accuracy: 98.71% 32 Validation loss: 0.069520 Best loss: 0.042150 Accuracy: 98.87% 33 Validation loss: 0.074562 Best loss: 0.042150 Accuracy: 98.71% 34 Validation loss: 0.116793 Best loss: 0.042150 Accuracy: 98.55% 35 Validation loss: 0.142083 Best loss: 0.042150 Accuracy: 98.28% 36 Validation loss: 0.148135 Best loss: 0.042150 Accuracy: 98.67% 37 Validation loss: 0.256867 Best loss: 0.042150 Accuracy: 97.69% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.6min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.125956 Best loss: 0.125956 Accuracy: 96.21% 1 Validation loss: 0.094917 Best loss: 0.094917 Accuracy: 97.46% 2 Validation loss: 0.090277 Best loss: 0.090277 Accuracy: 97.81% 3 Validation loss: 0.062243 Best loss: 0.062243 Accuracy: 98.48% 4 Validation loss: 0.112602 Best loss: 0.062243 Accuracy: 97.22% 5 Validation loss: 0.142495 Best loss: 0.062243 Accuracy: 97.07% 6 Validation loss: 0.091398 Best loss: 0.062243 Accuracy: 97.85% 7 Validation loss: 0.068427 Best loss: 0.062243 Accuracy: 98.32% 8 Validation loss: 0.044539 Best loss: 0.044539 Accuracy: 98.63% 9 Validation loss: 0.071943 Best loss: 0.044539 Accuracy: 98.24% 10 Validation loss: 0.251579 Best loss: 0.044539 Accuracy: 96.68% 11 Validation loss: 0.188345 Best loss: 0.044539 Accuracy: 97.65% 12 Validation loss: 0.073518 Best loss: 0.044539 Accuracy: 98.40% 13 Validation loss: 0.050502 Best loss: 0.044539 Accuracy: 98.67% 14 Validation loss: 0.060780 Best loss: 0.044539 Accuracy: 98.71% 15 Validation loss: 0.067121 Best loss: 0.044539 Accuracy: 98.40% 16 Validation loss: 0.060108 Best loss: 0.044539 Accuracy: 98.44% 17 Validation loss: 0.116106 Best loss: 0.044539 Accuracy: 97.81% 18 Validation loss: 0.130306 Best loss: 0.044539 Accuracy: 98.08% 19 Validation loss: 0.091272 Best loss: 0.044539 Accuracy: 98.44% 20 Validation loss: 0.075988 Best loss: 0.044539 Accuracy: 98.48% 21 Validation loss: 0.099954 Best loss: 0.044539 Accuracy: 98.32% 22 Validation loss: 0.112202 Best loss: 0.044539 Accuracy: 98.71% 23 Validation loss: 0.063877 Best loss: 0.044539 Accuracy: 98.75% 24 Validation loss: 0.124440 Best loss: 0.044539 Accuracy: 98.48% 25 Validation loss: 0.066578 Best loss: 0.044539 Accuracy: 98.71% 26 Validation loss: 0.102383 Best loss: 0.044539 Accuracy: 98.28% 27 Validation loss: 0.102578 Best loss: 0.044539 Accuracy: 98.44% 28 Validation loss: 0.077723 Best loss: 0.044539 Accuracy: 98.91% 29 Validation loss: 0.062676 Best loss: 0.044539 Accuracy: 99.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.142919 Best loss: 0.142919 Accuracy: 96.40% 1 Validation loss: 0.134700 Best loss: 0.134700 Accuracy: 96.52% 2 Validation loss: 0.074392 Best loss: 0.074392 Accuracy: 97.81% 3 Validation loss: 0.068717 Best loss: 0.068717 Accuracy: 98.32% 4 Validation loss: 0.087754 Best loss: 0.068717 Accuracy: 98.08% 5 Validation loss: 0.084656 Best loss: 0.068717 Accuracy: 97.77% 6 Validation loss: 0.424339 Best loss: 0.068717 Accuracy: 89.64% 7 Validation loss: 0.100895 Best loss: 0.068717 Accuracy: 97.89% 8 Validation loss: 0.059629 Best loss: 0.059629 Accuracy: 98.55% 9 Validation loss: 0.100412 Best loss: 0.059629 Accuracy: 97.93% 10 Validation loss: 0.076502 Best loss: 0.059629 Accuracy: 98.28% 11 Validation loss: 0.059201 Best loss: 0.059201 Accuracy: 98.75% 12 Validation loss: 0.059007 Best loss: 0.059007 Accuracy: 98.63% 13 Validation loss: 0.065247 Best loss: 0.059007 Accuracy: 98.71% 14 Validation loss: 0.210136 Best loss: 0.059007 Accuracy: 94.76% 15 Validation loss: 0.093718 Best loss: 0.059007 Accuracy: 98.48% 16 Validation loss: 0.046688 Best loss: 0.046688 Accuracy: 98.63% 17 Validation loss: 0.052756 Best loss: 0.046688 Accuracy: 98.71% 18 Validation loss: 0.068903 Best loss: 0.046688 Accuracy: 98.55% 19 Validation loss: 0.061866 Best loss: 0.046688 Accuracy: 98.75% 20 Validation loss: 0.055947 Best loss: 0.046688 Accuracy: 98.67% 21 Validation loss: 0.140336 Best loss: 0.046688 Accuracy: 98.36% 22 Validation loss: 0.077991 Best loss: 0.046688 Accuracy: 98.59% 23 Validation loss: 0.063328 Best loss: 0.046688 Accuracy: 98.87% 24 Validation loss: 0.079578 Best loss: 0.046688 Accuracy: 98.67% 25 Validation loss: 0.058026 Best loss: 0.046688 Accuracy: 98.87% 26 Validation loss: 0.085110 Best loss: 0.046688 Accuracy: 98.44% 27 Validation loss: 0.099225 Best loss: 0.046688 Accuracy: 98.83% 28 Validation loss: 0.176551 Best loss: 0.046688 Accuracy: 98.36% 29 Validation loss: 0.084755 Best loss: 0.046688 Accuracy: 98.63% 30 Validation loss: 0.075966 Best loss: 0.046688 Accuracy: 98.91% 31 Validation loss: 0.077510 Best loss: 0.046688 Accuracy: 99.06% 32 Validation loss: 0.095229 Best loss: 0.046688 Accuracy: 98.87% 33 Validation loss: 0.189773 Best loss: 0.046688 Accuracy: 98.67% 34 Validation loss: 0.113383 Best loss: 0.046688 Accuracy: 98.83% 35 Validation loss: 0.070859 Best loss: 0.046688 Accuracy: 99.06% 36 Validation loss: 0.067516 Best loss: 0.046688 Accuracy: 99.02% 37 Validation loss: 0.074743 Best loss: 0.046688 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.6min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.089046 Best loss: 0.089046 Accuracy: 97.58% 1 Validation loss: 0.072691 Best loss: 0.072691 Accuracy: 98.16% 2 Validation loss: 0.055336 Best loss: 0.055336 Accuracy: 98.36% 3 Validation loss: 0.051492 Best loss: 0.051492 Accuracy: 98.24% 4 Validation loss: 0.041475 Best loss: 0.041475 Accuracy: 98.79% 5 Validation loss: 0.050415 Best loss: 0.041475 Accuracy: 98.75% 6 Validation loss: 0.038674 Best loss: 0.038674 Accuracy: 98.87% 7 Validation loss: 0.047245 Best loss: 0.038674 Accuracy: 98.55% 8 Validation loss: 0.166988 Best loss: 0.038674 Accuracy: 97.58% 9 Validation loss: 0.042529 Best loss: 0.038674 Accuracy: 98.83% 10 Validation loss: 0.045867 Best loss: 0.038674 Accuracy: 98.94% 11 Validation loss: 0.058015 Best loss: 0.038674 Accuracy: 98.44% 12 Validation loss: 0.051672 Best loss: 0.038674 Accuracy: 98.83% 13 Validation loss: 0.050520 Best loss: 0.038674 Accuracy: 98.83% 14 Validation loss: 0.039948 Best loss: 0.038674 Accuracy: 99.02% 15 Validation loss: 0.055675 Best loss: 0.038674 Accuracy: 98.71% 16 Validation loss: 0.040251 Best loss: 0.038674 Accuracy: 98.79% 17 Validation loss: 0.040803 Best loss: 0.038674 Accuracy: 98.79% 18 Validation loss: 0.036056 Best loss: 0.036056 Accuracy: 99.06% 19 Validation loss: 0.056062 Best loss: 0.036056 Accuracy: 98.83% 20 Validation loss: 0.044380 Best loss: 0.036056 Accuracy: 98.98% 21 Validation loss: 0.065032 Best loss: 0.036056 Accuracy: 98.67% 22 Validation loss: 0.042281 Best loss: 0.036056 Accuracy: 99.02% 23 Validation loss: 0.054123 Best loss: 0.036056 Accuracy: 99.06% 24 Validation loss: 0.040435 Best loss: 0.036056 Accuracy: 99.06% 25 Validation loss: 0.066360 Best loss: 0.036056 Accuracy: 98.59% 26 Validation loss: 0.053802 Best loss: 0.036056 Accuracy: 98.71% 27 Validation loss: 0.048938 Best loss: 0.036056 Accuracy: 98.94% 28 Validation loss: 0.049807 Best loss: 0.036056 Accuracy: 98.91% 29 Validation loss: 0.041811 Best loss: 0.036056 Accuracy: 99.14% 30 Validation loss: 0.043563 Best loss: 0.036056 Accuracy: 99.10% 31 Validation loss: 0.048479 Best loss: 0.036056 Accuracy: 99.06% 32 Validation loss: 0.057982 Best loss: 0.036056 Accuracy: 98.87% 33 Validation loss: 0.052456 Best loss: 0.036056 Accuracy: 99.22% 34 Validation loss: 0.078002 Best loss: 0.036056 Accuracy: 98.67% 35 Validation loss: 0.051475 Best loss: 0.036056 Accuracy: 98.98% 36 Validation loss: 0.059964 Best loss: 0.036056 Accuracy: 98.75% 37 Validation loss: 0.053168 Best loss: 0.036056 Accuracy: 98.94% 38 Validation loss: 0.042752 Best loss: 0.036056 Accuracy: 98.98% 39 Validation loss: 0.035335 Best loss: 0.035335 Accuracy: 99.10% 40 Validation loss: 0.040239 Best loss: 0.035335 Accuracy: 99.14% 41 Validation loss: 0.054714 Best loss: 0.035335 Accuracy: 98.98% 42 Validation loss: 0.045566 Best loss: 0.035335 Accuracy: 99.06% 43 Validation loss: 0.050199 Best loss: 0.035335 Accuracy: 98.87% 44 Validation loss: 0.055576 Best loss: 0.035335 Accuracy: 98.87% 45 Validation loss: 0.046321 Best loss: 0.035335 Accuracy: 98.98% 46 Validation loss: 0.056516 Best loss: 0.035335 Accuracy: 98.87% 47 Validation loss: 0.048012 Best loss: 0.035335 Accuracy: 99.06% 48 Validation loss: 0.042314 Best loss: 0.035335 Accuracy: 99.26% 49 Validation loss: 0.051735 Best loss: 0.035335 Accuracy: 99.06% 50 Validation loss: 0.064683 Best loss: 0.035335 Accuracy: 98.91% 51 Validation loss: 0.055100 Best loss: 0.035335 Accuracy: 98.94% 52 Validation loss: 0.055443 Best loss: 0.035335 Accuracy: 99.02% 53 Validation loss: 0.069203 Best loss: 0.035335 Accuracy: 98.79% 54 Validation loss: 0.053085 Best loss: 0.035335 Accuracy: 99.06% 55 Validation loss: 0.052146 Best loss: 0.035335 Accuracy: 99.02% 56 Validation loss: 0.060151 Best loss: 0.035335 Accuracy: 99.14% 57 Validation loss: 0.064660 Best loss: 0.035335 Accuracy: 98.91% 58 Validation loss: 0.055981 Best loss: 0.035335 Accuracy: 98.91% 59 Validation loss: 0.057247 Best loss: 0.035335 Accuracy: 99.02% 60 Validation loss: 0.043760 Best loss: 0.035335 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02, total= 2.6min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.126155 Best loss: 0.126155 Accuracy: 96.48% 1 Validation loss: 0.059205 Best loss: 0.059205 Accuracy: 98.51% 2 Validation loss: 0.059315 Best loss: 0.059205 Accuracy: 98.32% 3 Validation loss: 0.072694 Best loss: 0.059205 Accuracy: 98.12% 4 Validation loss: 0.052085 Best loss: 0.052085 Accuracy: 98.51% 5 Validation loss: 0.070533 Best loss: 0.052085 Accuracy: 98.08% 6 Validation loss: 0.065484 Best loss: 0.052085 Accuracy: 98.24% 7 Validation loss: 0.044598 Best loss: 0.044598 Accuracy: 98.75% 8 Validation loss: 0.047897 Best loss: 0.044598 Accuracy: 98.79% 9 Validation loss: 0.050730 Best loss: 0.044598 Accuracy: 98.71% 10 Validation loss: 0.050214 Best loss: 0.044598 Accuracy: 98.91% 11 Validation loss: 0.040273 Best loss: 0.040273 Accuracy: 98.79% 12 Validation loss: 0.047540 Best loss: 0.040273 Accuracy: 98.91% 13 Validation loss: 0.057563 Best loss: 0.040273 Accuracy: 98.63% 14 Validation loss: 0.043323 Best loss: 0.040273 Accuracy: 98.75% 15 Validation loss: 0.076522 Best loss: 0.040273 Accuracy: 98.20% 16 Validation loss: 0.050215 Best loss: 0.040273 Accuracy: 98.79% 17 Validation loss: 0.055418 Best loss: 0.040273 Accuracy: 98.63% 18 Validation loss: 0.070927 Best loss: 0.040273 Accuracy: 98.20% 19 Validation loss: 0.047514 Best loss: 0.040273 Accuracy: 98.94% 20 Validation loss: 0.049006 Best loss: 0.040273 Accuracy: 98.79% 21 Validation loss: 0.037421 Best loss: 0.037421 Accuracy: 99.10% 22 Validation loss: 0.064827 Best loss: 0.037421 Accuracy: 98.87% 23 Validation loss: 0.047028 Best loss: 0.037421 Accuracy: 98.87% 24 Validation loss: 0.038679 Best loss: 0.037421 Accuracy: 99.10% 25 Validation loss: 0.041955 Best loss: 0.037421 Accuracy: 98.91% 26 Validation loss: 0.040104 Best loss: 0.037421 Accuracy: 98.98% 27 Validation loss: 0.044394 Best loss: 0.037421 Accuracy: 98.98% 28 Validation loss: 0.052222 Best loss: 0.037421 Accuracy: 98.91% 29 Validation loss: 0.042342 Best loss: 0.037421 Accuracy: 99.10% 30 Validation loss: 0.045150 Best loss: 0.037421 Accuracy: 99.10% 31 Validation loss: 0.060776 Best loss: 0.037421 Accuracy: 98.71% 32 Validation loss: 0.055974 Best loss: 0.037421 Accuracy: 98.75% 33 Validation loss: 0.030036 Best loss: 0.030036 Accuracy: 99.10% 34 Validation loss: 0.051309 Best loss: 0.030036 Accuracy: 98.94% 35 Validation loss: 0.040874 Best loss: 0.030036 Accuracy: 98.98% 36 Validation loss: 0.052250 Best loss: 0.030036 Accuracy: 98.83% 37 Validation loss: 0.043629 Best loss: 0.030036 Accuracy: 99.10% 38 Validation loss: 0.057504 Best loss: 0.030036 Accuracy: 98.91% 39 Validation loss: 0.040309 Best loss: 0.030036 Accuracy: 98.91% 40 Validation loss: 0.042893 Best loss: 0.030036 Accuracy: 99.06% 41 Validation loss: 0.048869 Best loss: 0.030036 Accuracy: 99.02% 42 Validation loss: 0.084335 Best loss: 0.030036 Accuracy: 98.71% 43 Validation loss: 0.037927 Best loss: 0.030036 Accuracy: 99.37% 44 Validation loss: 0.044251 Best loss: 0.030036 Accuracy: 99.18% 45 Validation loss: 0.047206 Best loss: 0.030036 Accuracy: 99.30% 46 Validation loss: 0.038641 Best loss: 0.030036 Accuracy: 99.26% 47 Validation loss: 0.051135 Best loss: 0.030036 Accuracy: 99.14% 48 Validation loss: 0.037729 Best loss: 0.030036 Accuracy: 99.30% 49 Validation loss: 0.040263 Best loss: 0.030036 Accuracy: 99.26% 50 Validation loss: 0.055763 Best loss: 0.030036 Accuracy: 99.22% 51 Validation loss: 0.044914 Best loss: 0.030036 Accuracy: 99.06% 52 Validation loss: 0.041934 Best loss: 0.030036 Accuracy: 98.94% 53 Validation loss: 0.056299 Best loss: 0.030036 Accuracy: 98.79% 54 Validation loss: 0.043354 Best loss: 0.030036 Accuracy: 99.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02, total= 2.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.088607 Best loss: 0.088607 Accuracy: 97.77% 1 Validation loss: 0.059020 Best loss: 0.059020 Accuracy: 98.16% 2 Validation loss: 0.057114 Best loss: 0.057114 Accuracy: 98.08% 3 Validation loss: 0.059019 Best loss: 0.057114 Accuracy: 98.16% 4 Validation loss: 0.061636 Best loss: 0.057114 Accuracy: 98.40% 5 Validation loss: 0.058661 Best loss: 0.057114 Accuracy: 98.24% 6 Validation loss: 0.042369 Best loss: 0.042369 Accuracy: 98.71% 7 Validation loss: 0.048458 Best loss: 0.042369 Accuracy: 98.63% 8 Validation loss: 0.049846 Best loss: 0.042369 Accuracy: 98.91% 9 Validation loss: 0.042974 Best loss: 0.042369 Accuracy: 98.83% 10 Validation loss: 0.055856 Best loss: 0.042369 Accuracy: 98.71% 11 Validation loss: 0.044446 Best loss: 0.042369 Accuracy: 98.75% 12 Validation loss: 0.048223 Best loss: 0.042369 Accuracy: 98.83% 13 Validation loss: 0.045407 Best loss: 0.042369 Accuracy: 98.67% 14 Validation loss: 0.047631 Best loss: 0.042369 Accuracy: 98.94% 15 Validation loss: 0.038329 Best loss: 0.038329 Accuracy: 99.06% 16 Validation loss: 0.035511 Best loss: 0.035511 Accuracy: 98.94% 17 Validation loss: 0.040064 Best loss: 0.035511 Accuracy: 99.10% 18 Validation loss: 0.042775 Best loss: 0.035511 Accuracy: 99.02% 19 Validation loss: 0.038636 Best loss: 0.035511 Accuracy: 99.30% 20 Validation loss: 0.039517 Best loss: 0.035511 Accuracy: 99.02% 21 Validation loss: 0.073186 Best loss: 0.035511 Accuracy: 98.36% 22 Validation loss: 0.040380 Best loss: 0.035511 Accuracy: 99.06% 23 Validation loss: 0.040062 Best loss: 0.035511 Accuracy: 99.06% 24 Validation loss: 0.056210 Best loss: 0.035511 Accuracy: 98.91% 25 Validation loss: 0.037450 Best loss: 0.035511 Accuracy: 99.22% 26 Validation loss: 0.046155 Best loss: 0.035511 Accuracy: 98.75% 27 Validation loss: 0.037171 Best loss: 0.035511 Accuracy: 99.22% 28 Validation loss: 0.040529 Best loss: 0.035511 Accuracy: 99.22% 29 Validation loss: 0.044966 Best loss: 0.035511 Accuracy: 99.06% 30 Validation loss: 0.060305 Best loss: 0.035511 Accuracy: 98.91% 31 Validation loss: 0.052044 Best loss: 0.035511 Accuracy: 98.98% 32 Validation loss: 0.050210 Best loss: 0.035511 Accuracy: 98.98% 33 Validation loss: 0.042219 Best loss: 0.035511 Accuracy: 99.10% 34 Validation loss: 0.049369 Best loss: 0.035511 Accuracy: 99.02% 35 Validation loss: 0.036546 Best loss: 0.035511 Accuracy: 99.18% 36 Validation loss: 0.022339 Best loss: 0.022339 Accuracy: 99.37% 37 Validation loss: 0.035007 Best loss: 0.022339 Accuracy: 99.22% 38 Validation loss: 0.053559 Best loss: 0.022339 Accuracy: 98.91% 39 Validation loss: 0.042837 Best loss: 0.022339 Accuracy: 99.06% 40 Validation loss: 0.030863 Best loss: 0.022339 Accuracy: 99.14% 41 Validation loss: 0.037139 Best loss: 0.022339 Accuracy: 99.18% 42 Validation loss: 0.049128 Best loss: 0.022339 Accuracy: 98.83% 43 Validation loss: 0.037604 Best loss: 0.022339 Accuracy: 98.98% 44 Validation loss: 0.052104 Best loss: 0.022339 Accuracy: 98.79% 45 Validation loss: 0.036232 Best loss: 0.022339 Accuracy: 99.10% 46 Validation loss: 0.039322 Best loss: 0.022339 Accuracy: 99.14% 47 Validation loss: 0.044262 Best loss: 0.022339 Accuracy: 99.02% 48 Validation loss: 0.044755 Best loss: 0.022339 Accuracy: 99.14% 49 Validation loss: 0.036428 Best loss: 0.022339 Accuracy: 99.30% 50 Validation loss: 0.028770 Best loss: 0.022339 Accuracy: 99.37% 51 Validation loss: 0.037347 Best loss: 0.022339 Accuracy: 99.37% 52 Validation loss: 0.048082 Best loss: 0.022339 Accuracy: 99.02% 53 Validation loss: 0.030448 Best loss: 0.022339 Accuracy: 99.30% 54 Validation loss: 0.043470 Best loss: 0.022339 Accuracy: 99.34% 55 Validation loss: 0.038348 Best loss: 0.022339 Accuracy: 99.10% 56 Validation loss: 0.032679 Best loss: 0.022339 Accuracy: 99.14% 57 Validation loss: 0.041999 Best loss: 0.022339 Accuracy: 99.30% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=50, batch_norm_momentum=0.98, learning_rate=0.02, total= 2.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.061712 Best loss: 0.061712 Accuracy: 98.20% 1 Validation loss: 0.060392 Best loss: 0.060392 Accuracy: 97.97% 2 Validation loss: 0.054341 Best loss: 0.054341 Accuracy: 98.63% 3 Validation loss: 0.048040 Best loss: 0.048040 Accuracy: 98.83% 4 Validation loss: 0.039998 Best loss: 0.039998 Accuracy: 98.59% 5 Validation loss: 0.042639 Best loss: 0.039998 Accuracy: 98.87% 6 Validation loss: 0.038493 Best loss: 0.038493 Accuracy: 98.91% 7 Validation loss: 0.045134 Best loss: 0.038493 Accuracy: 98.67% 8 Validation loss: 0.039672 Best loss: 0.038493 Accuracy: 98.91% 9 Validation loss: 0.039754 Best loss: 0.038493 Accuracy: 98.83% 10 Validation loss: 0.038130 Best loss: 0.038130 Accuracy: 98.94% 11 Validation loss: 0.040557 Best loss: 0.038130 Accuracy: 98.91% 12 Validation loss: 0.042665 Best loss: 0.038130 Accuracy: 98.94% 13 Validation loss: 0.054362 Best loss: 0.038130 Accuracy: 98.79% 14 Validation loss: 0.055043 Best loss: 0.038130 Accuracy: 98.55% 15 Validation loss: 0.042845 Best loss: 0.038130 Accuracy: 98.79% 16 Validation loss: 0.048979 Best loss: 0.038130 Accuracy: 98.87% 17 Validation loss: 0.035502 Best loss: 0.035502 Accuracy: 99.10% 18 Validation loss: 0.043877 Best loss: 0.035502 Accuracy: 98.79% 19 Validation loss: 0.034953 Best loss: 0.034953 Accuracy: 99.18% 20 Validation loss: 0.049729 Best loss: 0.034953 Accuracy: 98.79% 21 Validation loss: 0.060562 Best loss: 0.034953 Accuracy: 98.75% 22 Validation loss: 0.044265 Best loss: 0.034953 Accuracy: 99.10% 23 Validation loss: 0.052050 Best loss: 0.034953 Accuracy: 98.94% 24 Validation loss: 0.046476 Best loss: 0.034953 Accuracy: 98.94% 25 Validation loss: 0.055574 Best loss: 0.034953 Accuracy: 98.83% 26 Validation loss: 0.040052 Best loss: 0.034953 Accuracy: 99.10% 27 Validation loss: 0.045079 Best loss: 0.034953 Accuracy: 99.02% 28 Validation loss: 0.049023 Best loss: 0.034953 Accuracy: 99.06% 29 Validation loss: 0.045878 Best loss: 0.034953 Accuracy: 99.10% 30 Validation loss: 0.045529 Best loss: 0.034953 Accuracy: 98.83% 31 Validation loss: 0.066908 Best loss: 0.034953 Accuracy: 98.94% 32 Validation loss: 0.065858 Best loss: 0.034953 Accuracy: 98.98% 33 Validation loss: 0.029873 Best loss: 0.029873 Accuracy: 99.14% 34 Validation loss: 0.047789 Best loss: 0.029873 Accuracy: 98.87% 35 Validation loss: 0.032906 Best loss: 0.029873 Accuracy: 99.10% 36 Validation loss: 0.031404 Best loss: 0.029873 Accuracy: 99.14% 37 Validation loss: 0.044023 Best loss: 0.029873 Accuracy: 98.83% 38 Validation loss: 0.032848 Best loss: 0.029873 Accuracy: 99.30% 39 Validation loss: 0.033121 Best loss: 0.029873 Accuracy: 99.14% 40 Validation loss: 0.047982 Best loss: 0.029873 Accuracy: 98.94% 41 Validation loss: 0.045578 Best loss: 0.029873 Accuracy: 98.91% 42 Validation loss: 0.041164 Best loss: 0.029873 Accuracy: 99.14% 43 Validation loss: 0.035701 Best loss: 0.029873 Accuracy: 99.30% 44 Validation loss: 0.046417 Best loss: 0.029873 Accuracy: 98.94% 45 Validation loss: 0.041832 Best loss: 0.029873 Accuracy: 98.83% 46 Validation loss: 0.036894 Best loss: 0.029873 Accuracy: 99.10% 47 Validation loss: 0.043191 Best loss: 0.029873 Accuracy: 98.87% 48 Validation loss: 0.043468 Best loss: 0.029873 Accuracy: 99.10% 49 Validation loss: 0.042232 Best loss: 0.029873 Accuracy: 99.06% 50 Validation loss: 0.043235 Best loss: 0.029873 Accuracy: 98.98% 51 Validation loss: 0.038631 Best loss: 0.029873 Accuracy: 99.18% 52 Validation loss: 0.037313 Best loss: 0.029873 Accuracy: 99.22% 53 Validation loss: 0.050151 Best loss: 0.029873 Accuracy: 99.10% 54 Validation loss: 0.053613 Best loss: 0.029873 Accuracy: 98.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.061620 Best loss: 0.061620 Accuracy: 98.08% 1 Validation loss: 0.049404 Best loss: 0.049404 Accuracy: 98.40% 2 Validation loss: 0.050841 Best loss: 0.049404 Accuracy: 98.48% 3 Validation loss: 0.049837 Best loss: 0.049404 Accuracy: 98.48% 4 Validation loss: 0.054431 Best loss: 0.049404 Accuracy: 98.67% 5 Validation loss: 0.038168 Best loss: 0.038168 Accuracy: 98.91% 6 Validation loss: 0.046040 Best loss: 0.038168 Accuracy: 98.75% 7 Validation loss: 0.042818 Best loss: 0.038168 Accuracy: 98.87% 8 Validation loss: 0.044754 Best loss: 0.038168 Accuracy: 99.02% 9 Validation loss: 0.041841 Best loss: 0.038168 Accuracy: 99.02% 10 Validation loss: 0.048616 Best loss: 0.038168 Accuracy: 98.94% 11 Validation loss: 0.045556 Best loss: 0.038168 Accuracy: 98.98% 12 Validation loss: 0.042856 Best loss: 0.038168 Accuracy: 99.02% 13 Validation loss: 0.039390 Best loss: 0.038168 Accuracy: 98.94% 14 Validation loss: 0.034716 Best loss: 0.034716 Accuracy: 99.26% 15 Validation loss: 0.051582 Best loss: 0.034716 Accuracy: 98.71% 16 Validation loss: 0.035453 Best loss: 0.034716 Accuracy: 98.71% 17 Validation loss: 0.031419 Best loss: 0.031419 Accuracy: 98.98% 18 Validation loss: 0.057715 Best loss: 0.031419 Accuracy: 98.55% 19 Validation loss: 0.050240 Best loss: 0.031419 Accuracy: 98.87% 20 Validation loss: 0.040978 Best loss: 0.031419 Accuracy: 98.87% 21 Validation loss: 0.053826 Best loss: 0.031419 Accuracy: 98.75% 22 Validation loss: 0.051504 Best loss: 0.031419 Accuracy: 98.91% 23 Validation loss: 0.049188 Best loss: 0.031419 Accuracy: 98.94% 24 Validation loss: 0.032197 Best loss: 0.031419 Accuracy: 99.10% 25 Validation loss: 0.035581 Best loss: 0.031419 Accuracy: 99.18% 26 Validation loss: 0.034326 Best loss: 0.031419 Accuracy: 99.22% 27 Validation loss: 0.049305 Best loss: 0.031419 Accuracy: 98.91% 28 Validation loss: 0.048366 Best loss: 0.031419 Accuracy: 98.71% 29 Validation loss: 0.036926 Best loss: 0.031419 Accuracy: 99.26% 30 Validation loss: 0.041111 Best loss: 0.031419 Accuracy: 98.98% 31 Validation loss: 0.070886 Best loss: 0.031419 Accuracy: 98.75% 32 Validation loss: 0.059124 Best loss: 0.031419 Accuracy: 98.98% 33 Validation loss: 0.053165 Best loss: 0.031419 Accuracy: 99.18% 34 Validation loss: 0.046331 Best loss: 0.031419 Accuracy: 98.98% 35 Validation loss: 0.046262 Best loss: 0.031419 Accuracy: 99.06% 36 Validation loss: 0.053496 Best loss: 0.031419 Accuracy: 98.94% 37 Validation loss: 0.035710 Best loss: 0.031419 Accuracy: 99.02% 38 Validation loss: 0.037947 Best loss: 0.031419 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01, total= 54.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.058791 Best loss: 0.058791 Accuracy: 98.32% 1 Validation loss: 0.046576 Best loss: 0.046576 Accuracy: 98.48% 2 Validation loss: 0.050878 Best loss: 0.046576 Accuracy: 98.28% 3 Validation loss: 0.042208 Best loss: 0.042208 Accuracy: 98.63% 4 Validation loss: 0.037702 Best loss: 0.037702 Accuracy: 98.83% 5 Validation loss: 0.031859 Best loss: 0.031859 Accuracy: 99.06% 6 Validation loss: 0.029488 Best loss: 0.029488 Accuracy: 99.10% 7 Validation loss: 0.058740 Best loss: 0.029488 Accuracy: 98.24% 8 Validation loss: 0.043591 Best loss: 0.029488 Accuracy: 98.79% 9 Validation loss: 0.032071 Best loss: 0.029488 Accuracy: 99.22% 10 Validation loss: 0.046764 Best loss: 0.029488 Accuracy: 98.91% 11 Validation loss: 0.033919 Best loss: 0.029488 Accuracy: 99.06% 12 Validation loss: 0.039401 Best loss: 0.029488 Accuracy: 99.02% 13 Validation loss: 0.052074 Best loss: 0.029488 Accuracy: 98.71% 14 Validation loss: 0.036364 Best loss: 0.029488 Accuracy: 99.02% 15 Validation loss: 0.033067 Best loss: 0.029488 Accuracy: 99.10% 16 Validation loss: 0.037052 Best loss: 0.029488 Accuracy: 99.18% 17 Validation loss: 0.028077 Best loss: 0.028077 Accuracy: 99.14% 18 Validation loss: 0.032696 Best loss: 0.028077 Accuracy: 99.14% 19 Validation loss: 0.035398 Best loss: 0.028077 Accuracy: 98.94% 20 Validation loss: 0.040580 Best loss: 0.028077 Accuracy: 98.98% 21 Validation loss: 0.051306 Best loss: 0.028077 Accuracy: 98.67% 22 Validation loss: 0.035233 Best loss: 0.028077 Accuracy: 99.22% 23 Validation loss: 0.034053 Best loss: 0.028077 Accuracy: 99.02% 24 Validation loss: 0.050211 Best loss: 0.028077 Accuracy: 99.06% 25 Validation loss: 0.036947 Best loss: 0.028077 Accuracy: 98.98% 26 Validation loss: 0.028458 Best loss: 0.028077 Accuracy: 99.22% 27 Validation loss: 0.042643 Best loss: 0.028077 Accuracy: 98.87% 28 Validation loss: 0.031291 Best loss: 0.028077 Accuracy: 99.14% 29 Validation loss: 0.032563 Best loss: 0.028077 Accuracy: 98.94% 30 Validation loss: 0.033059 Best loss: 0.028077 Accuracy: 99.30% 31 Validation loss: 0.049366 Best loss: 0.028077 Accuracy: 99.06% 32 Validation loss: 0.037210 Best loss: 0.028077 Accuracy: 99.22% 33 Validation loss: 0.037784 Best loss: 0.028077 Accuracy: 99.14% 34 Validation loss: 0.040342 Best loss: 0.028077 Accuracy: 99.10% 35 Validation loss: 0.069757 Best loss: 0.028077 Accuracy: 98.67% 36 Validation loss: 0.033977 Best loss: 0.028077 Accuracy: 99.22% 37 Validation loss: 0.044085 Best loss: 0.028077 Accuracy: 98.87% 38 Validation loss: 0.046169 Best loss: 0.028077 Accuracy: 99.06% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=100, batch_norm_momentum=0.9, learning_rate=0.01, total= 54.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 4.836256 Best loss: 4.836256 Accuracy: 90.89% 1 Validation loss: 0.598753 Best loss: 0.598753 Accuracy: 94.57% 2 Validation loss: 0.235842 Best loss: 0.235842 Accuracy: 96.56% 3 Validation loss: 0.141703 Best loss: 0.141703 Accuracy: 96.91% 4 Validation loss: 0.154485 Best loss: 0.141703 Accuracy: 96.68% 5 Validation loss: 0.124688 Best loss: 0.124688 Accuracy: 97.34% 6 Validation loss: 0.087364 Best loss: 0.087364 Accuracy: 98.20% 7 Validation loss: 0.060215 Best loss: 0.060215 Accuracy: 98.67% 8 Validation loss: 0.063890 Best loss: 0.060215 Accuracy: 98.51% 9 Validation loss: 0.072716 Best loss: 0.060215 Accuracy: 98.63% 10 Validation loss: 0.095894 Best loss: 0.060215 Accuracy: 98.36% 11 Validation loss: 0.081026 Best loss: 0.060215 Accuracy: 98.59% 12 Validation loss: 0.082191 Best loss: 0.060215 Accuracy: 98.55% 13 Validation loss: 0.078633 Best loss: 0.060215 Accuracy: 98.36% 14 Validation loss: 0.077939 Best loss: 0.060215 Accuracy: 98.51% 15 Validation loss: 0.076284 Best loss: 0.060215 Accuracy: 98.48% 16 Validation loss: 0.057202 Best loss: 0.057202 Accuracy: 98.83% 17 Validation loss: 0.063481 Best loss: 0.057202 Accuracy: 98.59% 18 Validation loss: 0.074907 Best loss: 0.057202 Accuracy: 98.48% 19 Validation loss: 0.069796 Best loss: 0.057202 Accuracy: 98.67% 20 Validation loss: 0.044758 Best loss: 0.044758 Accuracy: 98.98% 21 Validation loss: 0.062930 Best loss: 0.044758 Accuracy: 98.83% 22 Validation loss: 0.068160 Best loss: 0.044758 Accuracy: 98.79% 23 Validation loss: 0.067989 Best loss: 0.044758 Accuracy: 98.63% 24 Validation loss: 0.061554 Best loss: 0.044758 Accuracy: 98.94% 25 Validation loss: 0.066501 Best loss: 0.044758 Accuracy: 98.75% 26 Validation loss: 0.079405 Best loss: 0.044758 Accuracy: 98.59% 27 Validation loss: 0.100535 Best loss: 0.044758 Accuracy: 98.28% 28 Validation loss: 0.095924 Best loss: 0.044758 Accuracy: 98.28% 29 Validation loss: 0.095848 Best loss: 0.044758 Accuracy: 98.24% 30 Validation loss: 0.093213 Best loss: 0.044758 Accuracy: 98.44% 31 Validation loss: 0.070028 Best loss: 0.044758 Accuracy: 98.75% 32 Validation loss: 0.055754 Best loss: 0.044758 Accuracy: 99.06% 33 Validation loss: 0.119830 Best loss: 0.044758 Accuracy: 98.24% 34 Validation loss: 0.083498 Best loss: 0.044758 Accuracy: 98.40% 35 Validation loss: 0.060775 Best loss: 0.044758 Accuracy: 98.67% 36 Validation loss: 0.055359 Best loss: 0.044758 Accuracy: 99.02% 37 Validation loss: 0.065444 Best loss: 0.044758 Accuracy: 98.75% 38 Validation loss: 0.058979 Best loss: 0.044758 Accuracy: 98.83% 39 Validation loss: 0.062855 Best loss: 0.044758 Accuracy: 98.87% 40 Validation loss: 0.075719 Best loss: 0.044758 Accuracy: 98.71% 41 Validation loss: 0.109948 Best loss: 0.044758 Accuracy: 98.20% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05, total= 16.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 4.707267 Best loss: 4.707267 Accuracy: 89.80% 1 Validation loss: 1.767822 Best loss: 1.767822 Accuracy: 89.95% 2 Validation loss: 0.532535 Best loss: 0.532535 Accuracy: 94.72% 3 Validation loss: 0.136687 Best loss: 0.136687 Accuracy: 98.08% 4 Validation loss: 0.155387 Best loss: 0.136687 Accuracy: 97.81% 5 Validation loss: 0.106990 Best loss: 0.106990 Accuracy: 98.12% 6 Validation loss: 0.069396 Best loss: 0.069396 Accuracy: 98.44% 7 Validation loss: 0.085276 Best loss: 0.069396 Accuracy: 98.24% 8 Validation loss: 0.087579 Best loss: 0.069396 Accuracy: 98.44% 9 Validation loss: 0.083032 Best loss: 0.069396 Accuracy: 98.59% 10 Validation loss: 0.096772 Best loss: 0.069396 Accuracy: 98.32% 11 Validation loss: 0.057175 Best loss: 0.057175 Accuracy: 98.91% 12 Validation loss: 0.071862 Best loss: 0.057175 Accuracy: 98.48% 13 Validation loss: 0.056319 Best loss: 0.056319 Accuracy: 98.87% 14 Validation loss: 0.069359 Best loss: 0.056319 Accuracy: 98.40% 15 Validation loss: 0.082025 Best loss: 0.056319 Accuracy: 98.40% 16 Validation loss: 0.069482 Best loss: 0.056319 Accuracy: 98.55% 17 Validation loss: 0.049885 Best loss: 0.049885 Accuracy: 98.98% 18 Validation loss: 0.093100 Best loss: 0.049885 Accuracy: 98.28% 19 Validation loss: 0.057117 Best loss: 0.049885 Accuracy: 98.94% 20 Validation loss: 0.071259 Best loss: 0.049885 Accuracy: 98.75% 21 Validation loss: 0.053836 Best loss: 0.049885 Accuracy: 99.06% 22 Validation loss: 0.075194 Best loss: 0.049885 Accuracy: 98.63% 23 Validation loss: 0.061668 Best loss: 0.049885 Accuracy: 98.83% 24 Validation loss: 0.047636 Best loss: 0.047636 Accuracy: 99.26% 25 Validation loss: 0.070596 Best loss: 0.047636 Accuracy: 98.94% 26 Validation loss: 0.062105 Best loss: 0.047636 Accuracy: 99.02% 27 Validation loss: 0.056679 Best loss: 0.047636 Accuracy: 98.83% 28 Validation loss: 0.050878 Best loss: 0.047636 Accuracy: 99.06% 29 Validation loss: 0.074268 Best loss: 0.047636 Accuracy: 98.91% 30 Validation loss: 0.057167 Best loss: 0.047636 Accuracy: 98.98% 31 Validation loss: 0.062324 Best loss: 0.047636 Accuracy: 99.06% 32 Validation loss: 0.095189 Best loss: 0.047636 Accuracy: 98.63% 33 Validation loss: 0.092739 Best loss: 0.047636 Accuracy: 98.55% 34 Validation loss: 0.062219 Best loss: 0.047636 Accuracy: 99.02% 35 Validation loss: 0.080036 Best loss: 0.047636 Accuracy: 98.48% 36 Validation loss: 0.060451 Best loss: 0.047636 Accuracy: 99.14% 37 Validation loss: 0.078941 Best loss: 0.047636 Accuracy: 98.75% 38 Validation loss: 0.060529 Best loss: 0.047636 Accuracy: 99.14% 39 Validation loss: 0.062054 Best loss: 0.047636 Accuracy: 98.87% 40 Validation loss: 0.059120 Best loss: 0.047636 Accuracy: 98.98% 41 Validation loss: 0.066834 Best loss: 0.047636 Accuracy: 98.63% 42 Validation loss: 0.085034 Best loss: 0.047636 Accuracy: 98.55% 43 Validation loss: 0.076673 Best loss: 0.047636 Accuracy: 98.59% 44 Validation loss: 0.083316 Best loss: 0.047636 Accuracy: 98.51% 45 Validation loss: 0.073503 Best loss: 0.047636 Accuracy: 98.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05, total= 16.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 6.919957 Best loss: 6.919957 Accuracy: 80.30% 1 Validation loss: 0.468069 Best loss: 0.468069 Accuracy: 93.00% 2 Validation loss: 0.295540 Best loss: 0.295540 Accuracy: 94.80% 3 Validation loss: 0.210347 Best loss: 0.210347 Accuracy: 96.68% 4 Validation loss: 0.090193 Best loss: 0.090193 Accuracy: 98.32% 5 Validation loss: 0.105541 Best loss: 0.090193 Accuracy: 98.24% 6 Validation loss: 0.081325 Best loss: 0.081325 Accuracy: 98.55% 7 Validation loss: 0.071348 Best loss: 0.071348 Accuracy: 98.59% 8 Validation loss: 0.094898 Best loss: 0.071348 Accuracy: 98.44% 9 Validation loss: 0.097051 Best loss: 0.071348 Accuracy: 98.28% 10 Validation loss: 0.078046 Best loss: 0.071348 Accuracy: 98.36% 11 Validation loss: 0.084042 Best loss: 0.071348 Accuracy: 98.67% 12 Validation loss: 0.072908 Best loss: 0.071348 Accuracy: 98.94% 13 Validation loss: 0.075689 Best loss: 0.071348 Accuracy: 98.75% 14 Validation loss: 0.066885 Best loss: 0.066885 Accuracy: 98.91% 15 Validation loss: 0.066020 Best loss: 0.066020 Accuracy: 98.71% 16 Validation loss: 0.082090 Best loss: 0.066020 Accuracy: 98.44% 17 Validation loss: 0.081434 Best loss: 0.066020 Accuracy: 98.36% 18 Validation loss: 0.085077 Best loss: 0.066020 Accuracy: 98.55% 19 Validation loss: 0.084167 Best loss: 0.066020 Accuracy: 98.55% 20 Validation loss: 0.088668 Best loss: 0.066020 Accuracy: 98.67% 21 Validation loss: 0.069729 Best loss: 0.066020 Accuracy: 98.83% 22 Validation loss: 0.066986 Best loss: 0.066020 Accuracy: 98.83% 23 Validation loss: 0.081034 Best loss: 0.066020 Accuracy: 98.71% 24 Validation loss: 0.086464 Best loss: 0.066020 Accuracy: 98.79% 25 Validation loss: 0.084871 Best loss: 0.066020 Accuracy: 98.91% 26 Validation loss: 0.077318 Best loss: 0.066020 Accuracy: 98.63% 27 Validation loss: 0.077131 Best loss: 0.066020 Accuracy: 98.48% 28 Validation loss: 0.075721 Best loss: 0.066020 Accuracy: 98.67% 29 Validation loss: 0.078646 Best loss: 0.066020 Accuracy: 98.67% 30 Validation loss: 0.135781 Best loss: 0.066020 Accuracy: 97.73% 31 Validation loss: 0.062157 Best loss: 0.062157 Accuracy: 98.98% 32 Validation loss: 0.068585 Best loss: 0.062157 Accuracy: 99.06% 33 Validation loss: 0.083089 Best loss: 0.062157 Accuracy: 98.91% 34 Validation loss: 0.103073 Best loss: 0.062157 Accuracy: 98.67% 35 Validation loss: 0.079303 Best loss: 0.062157 Accuracy: 98.98% 36 Validation loss: 0.069185 Best loss: 0.062157 Accuracy: 98.79% 37 Validation loss: 0.064719 Best loss: 0.062157 Accuracy: 98.75% 38 Validation loss: 0.102800 Best loss: 0.062157 Accuracy: 98.67% 39 Validation loss: 0.075695 Best loss: 0.062157 Accuracy: 98.79% 40 Validation loss: 0.077575 Best loss: 0.062157 Accuracy: 98.71% 41 Validation loss: 0.058042 Best loss: 0.058042 Accuracy: 99.10% 42 Validation loss: 0.056664 Best loss: 0.056664 Accuracy: 99.02% 43 Validation loss: 0.064730 Best loss: 0.056664 Accuracy: 98.87% 44 Validation loss: 0.107633 Best loss: 0.056664 Accuracy: 98.36% 45 Validation loss: 0.073307 Best loss: 0.056664 Accuracy: 98.87% 46 Validation loss: 0.071340 Best loss: 0.056664 Accuracy: 98.79% 47 Validation loss: 0.071675 Best loss: 0.056664 Accuracy: 98.75% 48 Validation loss: 0.084380 Best loss: 0.056664 Accuracy: 98.63% 49 Validation loss: 0.102246 Best loss: 0.056664 Accuracy: 98.51% 50 Validation loss: 0.077002 Best loss: 0.056664 Accuracy: 98.75% 51 Validation loss: 0.070400 Best loss: 0.056664 Accuracy: 98.87% 52 Validation loss: 0.073437 Best loss: 0.056664 Accuracy: 98.87% 53 Validation loss: 0.072691 Best loss: 0.056664 Accuracy: 98.83% 54 Validation loss: 0.070807 Best loss: 0.056664 Accuracy: 98.87% 55 Validation loss: 0.077325 Best loss: 0.056664 Accuracy: 98.87% 56 Validation loss: 0.086305 Best loss: 0.056664 Accuracy: 98.55% 57 Validation loss: 0.076796 Best loss: 0.056664 Accuracy: 98.79% 58 Validation loss: 0.123355 Best loss: 0.056664 Accuracy: 98.48% 59 Validation loss: 0.065903 Best loss: 0.056664 Accuracy: 98.83% 60 Validation loss: 0.093622 Best loss: 0.056664 Accuracy: 98.98% 61 Validation loss: 0.099004 Best loss: 0.056664 Accuracy: 98.40% 62 Validation loss: 0.065218 Best loss: 0.056664 Accuracy: 98.79% 63 Validation loss: 0.071789 Best loss: 0.056664 Accuracy: 98.94% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=500, batch_norm_momentum=0.99, learning_rate=0.05, total= 22.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.125661 Best loss: 0.125661 Accuracy: 96.56% 1 Validation loss: 0.098061 Best loss: 0.098061 Accuracy: 96.83% 2 Validation loss: 0.063872 Best loss: 0.063872 Accuracy: 98.16% 3 Validation loss: 0.089017 Best loss: 0.063872 Accuracy: 97.46% 4 Validation loss: 0.054265 Best loss: 0.054265 Accuracy: 98.48% 5 Validation loss: 0.048222 Best loss: 0.048222 Accuracy: 98.63% 6 Validation loss: 0.049389 Best loss: 0.048222 Accuracy: 98.28% 7 Validation loss: 0.044232 Best loss: 0.044232 Accuracy: 98.63% 8 Validation loss: 0.044661 Best loss: 0.044232 Accuracy: 98.59% 9 Validation loss: 0.048843 Best loss: 0.044232 Accuracy: 98.28% 10 Validation loss: 0.048945 Best loss: 0.044232 Accuracy: 98.48% 11 Validation loss: 0.047802 Best loss: 0.044232 Accuracy: 98.55% 12 Validation loss: 0.047460 Best loss: 0.044232 Accuracy: 98.48% 13 Validation loss: 0.046249 Best loss: 0.044232 Accuracy: 98.71% 14 Validation loss: 0.048124 Best loss: 0.044232 Accuracy: 98.40% 15 Validation loss: 0.037900 Best loss: 0.037900 Accuracy: 98.79% 16 Validation loss: 0.041786 Best loss: 0.037900 Accuracy: 98.75% 17 Validation loss: 0.036832 Best loss: 0.036832 Accuracy: 98.71% 18 Validation loss: 0.041321 Best loss: 0.036832 Accuracy: 98.67% 19 Validation loss: 0.038101 Best loss: 0.036832 Accuracy: 98.79% 20 Validation loss: 0.041214 Best loss: 0.036832 Accuracy: 98.79% 21 Validation loss: 0.034587 Best loss: 0.034587 Accuracy: 99.02% 22 Validation loss: 0.043765 Best loss: 0.034587 Accuracy: 98.59% 23 Validation loss: 0.039373 Best loss: 0.034587 Accuracy: 98.94% 24 Validation loss: 0.039391 Best loss: 0.034587 Accuracy: 98.75% 25 Validation loss: 0.036165 Best loss: 0.034587 Accuracy: 98.91% 26 Validation loss: 0.037990 Best loss: 0.034587 Accuracy: 98.94% 27 Validation loss: 0.034863 Best loss: 0.034587 Accuracy: 98.91% 28 Validation loss: 0.047968 Best loss: 0.034587 Accuracy: 98.51% 29 Validation loss: 0.041035 Best loss: 0.034587 Accuracy: 98.71% 30 Validation loss: 0.041807 Best loss: 0.034587 Accuracy: 98.91% 31 Validation loss: 0.039881 Best loss: 0.034587 Accuracy: 98.79% 32 Validation loss: 0.037565 Best loss: 0.034587 Accuracy: 98.83% 33 Validation loss: 0.036728 Best loss: 0.034587 Accuracy: 98.67% 34 Validation loss: 0.034715 Best loss: 0.034587 Accuracy: 99.02% 35 Validation loss: 0.036321 Best loss: 0.034587 Accuracy: 98.94% 36 Validation loss: 0.038161 Best loss: 0.034587 Accuracy: 98.98% 37 Validation loss: 0.036534 Best loss: 0.034587 Accuracy: 98.83% 38 Validation loss: 0.034278 Best loss: 0.034278 Accuracy: 98.98% 39 Validation loss: 0.054729 Best loss: 0.034278 Accuracy: 98.71% 40 Validation loss: 0.039395 Best loss: 0.034278 Accuracy: 99.06% 41 Validation loss: 0.045315 Best loss: 0.034278 Accuracy: 98.67% 42 Validation loss: 0.038903 Best loss: 0.034278 Accuracy: 98.91% 43 Validation loss: 0.038716 Best loss: 0.034278 Accuracy: 98.83% 44 Validation loss: 0.033924 Best loss: 0.033924 Accuracy: 99.06% 45 Validation loss: 0.045033 Best loss: 0.033924 Accuracy: 98.67% 46 Validation loss: 0.040343 Best loss: 0.033924 Accuracy: 98.94% 47 Validation loss: 0.041511 Best loss: 0.033924 Accuracy: 98.91% 48 Validation loss: 0.057984 Best loss: 0.033924 Accuracy: 98.32% 49 Validation loss: 0.036368 Best loss: 0.033924 Accuracy: 98.87% 50 Validation loss: 0.027609 Best loss: 0.027609 Accuracy: 99.06% 51 Validation loss: 0.035406 Best loss: 0.027609 Accuracy: 98.91% 52 Validation loss: 0.047989 Best loss: 0.027609 Accuracy: 98.94% 53 Validation loss: 0.040391 Best loss: 0.027609 Accuracy: 98.83% 54 Validation loss: 0.041666 Best loss: 0.027609 Accuracy: 98.91% 55 Validation loss: 0.040892 Best loss: 0.027609 Accuracy: 98.94% 56 Validation loss: 0.040488 Best loss: 0.027609 Accuracy: 99.06% 57 Validation loss: 0.038031 Best loss: 0.027609 Accuracy: 98.98% 58 Validation loss: 0.035541 Best loss: 0.027609 Accuracy: 98.91% 59 Validation loss: 0.049448 Best loss: 0.027609 Accuracy: 98.67% 60 Validation loss: 0.039664 Best loss: 0.027609 Accuracy: 98.87% 61 Validation loss: 0.043431 Best loss: 0.027609 Accuracy: 98.87% 62 Validation loss: 0.055417 Best loss: 0.027609 Accuracy: 98.71% 63 Validation loss: 0.058818 Best loss: 0.027609 Accuracy: 98.48% 64 Validation loss: 0.051943 Best loss: 0.027609 Accuracy: 98.91% 65 Validation loss: 0.050482 Best loss: 0.027609 Accuracy: 98.59% 66 Validation loss: 0.047399 Best loss: 0.027609 Accuracy: 98.83% 67 Validation loss: 0.054120 Best loss: 0.027609 Accuracy: 98.75% 68 Validation loss: 0.054082 Best loss: 0.027609 Accuracy: 98.79% 69 Validation loss: 0.053655 Best loss: 0.027609 Accuracy: 98.75% 70 Validation loss: 0.053924 Best loss: 0.027609 Accuracy: 98.83% 71 Validation loss: 0.042294 Best loss: 0.027609 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total=12.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.103086 Best loss: 0.103086 Accuracy: 96.72% 1 Validation loss: 0.080791 Best loss: 0.080791 Accuracy: 97.30% 2 Validation loss: 0.077556 Best loss: 0.077556 Accuracy: 97.46% 3 Validation loss: 0.060100 Best loss: 0.060100 Accuracy: 97.69% 4 Validation loss: 0.060579 Best loss: 0.060100 Accuracy: 97.85% 5 Validation loss: 0.077110 Best loss: 0.060100 Accuracy: 97.65% 6 Validation loss: 0.058507 Best loss: 0.058507 Accuracy: 98.16% 7 Validation loss: 0.044345 Best loss: 0.044345 Accuracy: 98.67% 8 Validation loss: 0.049654 Best loss: 0.044345 Accuracy: 98.67% 9 Validation loss: 0.053307 Best loss: 0.044345 Accuracy: 98.28% 10 Validation loss: 0.044551 Best loss: 0.044345 Accuracy: 98.75% 11 Validation loss: 0.045347 Best loss: 0.044345 Accuracy: 98.63% 12 Validation loss: 0.037828 Best loss: 0.037828 Accuracy: 98.79% 13 Validation loss: 0.045012 Best loss: 0.037828 Accuracy: 98.67% 14 Validation loss: 0.043192 Best loss: 0.037828 Accuracy: 98.36% 15 Validation loss: 0.044648 Best loss: 0.037828 Accuracy: 98.67% 16 Validation loss: 0.064057 Best loss: 0.037828 Accuracy: 98.20% 17 Validation loss: 0.040607 Best loss: 0.037828 Accuracy: 98.67% 18 Validation loss: 0.042654 Best loss: 0.037828 Accuracy: 98.59% 19 Validation loss: 0.059392 Best loss: 0.037828 Accuracy: 98.12% 20 Validation loss: 0.057254 Best loss: 0.037828 Accuracy: 98.63% 21 Validation loss: 0.036607 Best loss: 0.036607 Accuracy: 98.94% 22 Validation loss: 0.039848 Best loss: 0.036607 Accuracy: 98.83% 23 Validation loss: 0.054527 Best loss: 0.036607 Accuracy: 98.51% 24 Validation loss: 0.041726 Best loss: 0.036607 Accuracy: 98.75% 25 Validation loss: 0.048718 Best loss: 0.036607 Accuracy: 98.55% 26 Validation loss: 0.044620 Best loss: 0.036607 Accuracy: 98.59% 27 Validation loss: 0.048967 Best loss: 0.036607 Accuracy: 98.28% 28 Validation loss: 0.041789 Best loss: 0.036607 Accuracy: 98.67% 29 Validation loss: 0.049716 Best loss: 0.036607 Accuracy: 98.91% 30 Validation loss: 0.068161 Best loss: 0.036607 Accuracy: 98.24% 31 Validation loss: 0.043282 Best loss: 0.036607 Accuracy: 98.83% 32 Validation loss: 0.044965 Best loss: 0.036607 Accuracy: 98.83% 33 Validation loss: 0.039039 Best loss: 0.036607 Accuracy: 98.75% 34 Validation loss: 0.040655 Best loss: 0.036607 Accuracy: 98.83% 35 Validation loss: 0.034081 Best loss: 0.034081 Accuracy: 98.79% 36 Validation loss: 0.054108 Best loss: 0.034081 Accuracy: 98.63% 37 Validation loss: 0.036368 Best loss: 0.034081 Accuracy: 99.02% 38 Validation loss: 0.042616 Best loss: 0.034081 Accuracy: 98.91% 39 Validation loss: 0.036898 Best loss: 0.034081 Accuracy: 98.83% 40 Validation loss: 0.042116 Best loss: 0.034081 Accuracy: 98.83% 41 Validation loss: 0.040657 Best loss: 0.034081 Accuracy: 98.83% 42 Validation loss: 0.047654 Best loss: 0.034081 Accuracy: 98.71% 43 Validation loss: 0.036818 Best loss: 0.034081 Accuracy: 98.87% 44 Validation loss: 0.042809 Best loss: 0.034081 Accuracy: 98.98% 45 Validation loss: 0.038417 Best loss: 0.034081 Accuracy: 98.83% 46 Validation loss: 0.053037 Best loss: 0.034081 Accuracy: 98.59% 47 Validation loss: 0.031427 Best loss: 0.031427 Accuracy: 99.10% 48 Validation loss: 0.038063 Best loss: 0.031427 Accuracy: 98.98% 49 Validation loss: 0.041364 Best loss: 0.031427 Accuracy: 98.91% 50 Validation loss: 0.044857 Best loss: 0.031427 Accuracy: 98.83% 51 Validation loss: 0.043814 Best loss: 0.031427 Accuracy: 98.83% 52 Validation loss: 0.045500 Best loss: 0.031427 Accuracy: 98.94% 53 Validation loss: 0.038333 Best loss: 0.031427 Accuracy: 98.87% 54 Validation loss: 0.046462 Best loss: 0.031427 Accuracy: 98.87% 55 Validation loss: 0.040134 Best loss: 0.031427 Accuracy: 98.83% 56 Validation loss: 0.040153 Best loss: 0.031427 Accuracy: 98.75% 57 Validation loss: 0.046202 Best loss: 0.031427 Accuracy: 98.83% 58 Validation loss: 0.047684 Best loss: 0.031427 Accuracy: 98.71% 59 Validation loss: 0.044877 Best loss: 0.031427 Accuracy: 98.91% 60 Validation loss: 0.035165 Best loss: 0.031427 Accuracy: 98.94% 61 Validation loss: 0.038858 Best loss: 0.031427 Accuracy: 98.91% 62 Validation loss: 0.045353 Best loss: 0.031427 Accuracy: 99.06% 63 Validation loss: 0.050662 Best loss: 0.031427 Accuracy: 98.44% 64 Validation loss: 0.050845 Best loss: 0.031427 Accuracy: 98.71% 65 Validation loss: 0.053246 Best loss: 0.031427 Accuracy: 98.59% 66 Validation loss: 0.040093 Best loss: 0.031427 Accuracy: 98.94% 67 Validation loss: 0.041264 Best loss: 0.031427 Accuracy: 98.98% 68 Validation loss: 0.038287 Best loss: 0.031427 Accuracy: 98.98% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total=11.7min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.085546 Best loss: 0.085546 Accuracy: 97.65% 1 Validation loss: 0.057352 Best loss: 0.057352 Accuracy: 98.05% 2 Validation loss: 0.069103 Best loss: 0.057352 Accuracy: 98.12% 3 Validation loss: 0.063647 Best loss: 0.057352 Accuracy: 97.81% 4 Validation loss: 0.060332 Best loss: 0.057352 Accuracy: 98.32% 5 Validation loss: 0.049603 Best loss: 0.049603 Accuracy: 98.24% 6 Validation loss: 0.057980 Best loss: 0.049603 Accuracy: 98.05% 7 Validation loss: 0.036549 Best loss: 0.036549 Accuracy: 98.75% 8 Validation loss: 0.058489 Best loss: 0.036549 Accuracy: 98.08% 9 Validation loss: 0.040028 Best loss: 0.036549 Accuracy: 98.48% 10 Validation loss: 0.038622 Best loss: 0.036549 Accuracy: 98.63% 11 Validation loss: 0.042649 Best loss: 0.036549 Accuracy: 98.55% 12 Validation loss: 0.045057 Best loss: 0.036549 Accuracy: 98.71% 13 Validation loss: 0.050367 Best loss: 0.036549 Accuracy: 98.44% 14 Validation loss: 0.043516 Best loss: 0.036549 Accuracy: 98.59% 15 Validation loss: 0.043465 Best loss: 0.036549 Accuracy: 98.75% 16 Validation loss: 0.040504 Best loss: 0.036549 Accuracy: 98.55% 17 Validation loss: 0.044459 Best loss: 0.036549 Accuracy: 98.83% 18 Validation loss: 0.036960 Best loss: 0.036549 Accuracy: 98.98% 19 Validation loss: 0.039070 Best loss: 0.036549 Accuracy: 98.79% 20 Validation loss: 0.038983 Best loss: 0.036549 Accuracy: 98.87% 21 Validation loss: 0.031094 Best loss: 0.031094 Accuracy: 98.87% 22 Validation loss: 0.028118 Best loss: 0.028118 Accuracy: 99.14% 23 Validation loss: 0.036910 Best loss: 0.028118 Accuracy: 98.87% 24 Validation loss: 0.033424 Best loss: 0.028118 Accuracy: 98.79% 25 Validation loss: 0.042049 Best loss: 0.028118 Accuracy: 98.75% 26 Validation loss: 0.034846 Best loss: 0.028118 Accuracy: 99.06% 27 Validation loss: 0.037225 Best loss: 0.028118 Accuracy: 98.83% 28 Validation loss: 0.026246 Best loss: 0.026246 Accuracy: 99.02% 29 Validation loss: 0.038921 Best loss: 0.026246 Accuracy: 98.75% 30 Validation loss: 0.047505 Best loss: 0.026246 Accuracy: 98.40% 31 Validation loss: 0.034842 Best loss: 0.026246 Accuracy: 98.63% 32 Validation loss: 0.028516 Best loss: 0.026246 Accuracy: 99.06% 33 Validation loss: 0.032243 Best loss: 0.026246 Accuracy: 98.94% 34 Validation loss: 0.031566 Best loss: 0.026246 Accuracy: 99.10% 35 Validation loss: 0.038633 Best loss: 0.026246 Accuracy: 98.75% 36 Validation loss: 0.038729 Best loss: 0.026246 Accuracy: 98.87% 37 Validation loss: 0.034450 Best loss: 0.026246 Accuracy: 98.94% 38 Validation loss: 0.036133 Best loss: 0.026246 Accuracy: 98.87% 39 Validation loss: 0.040653 Best loss: 0.026246 Accuracy: 98.83% 40 Validation loss: 0.035275 Best loss: 0.026246 Accuracy: 98.94% 41 Validation loss: 0.040575 Best loss: 0.026246 Accuracy: 98.91% 42 Validation loss: 0.036470 Best loss: 0.026246 Accuracy: 98.98% 43 Validation loss: 0.037797 Best loss: 0.026246 Accuracy: 99.02% 44 Validation loss: 0.033894 Best loss: 0.026246 Accuracy: 99.02% 45 Validation loss: 0.030858 Best loss: 0.026246 Accuracy: 99.06% 46 Validation loss: 0.038694 Best loss: 0.026246 Accuracy: 98.79% 47 Validation loss: 0.032697 Best loss: 0.026246 Accuracy: 98.91% 48 Validation loss: 0.033671 Best loss: 0.026246 Accuracy: 98.83% 49 Validation loss: 0.038719 Best loss: 0.026246 Accuracy: 98.71% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total= 8.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05 0 Validation loss: 0.161178 Best loss: 0.161178 Accuracy: 95.15% 1 Validation loss: 0.094390 Best loss: 0.094390 Accuracy: 97.42% 2 Validation loss: 0.266855 Best loss: 0.094390 Accuracy: 92.10% 3 Validation loss: 0.109986 Best loss: 0.094390 Accuracy: 96.79% 4 Validation loss: 0.066147 Best loss: 0.066147 Accuracy: 97.89% 5 Validation loss: 0.067305 Best loss: 0.066147 Accuracy: 98.12% 6 Validation loss: 0.055513 Best loss: 0.055513 Accuracy: 98.40% 7 Validation loss: 0.067701 Best loss: 0.055513 Accuracy: 98.36% 8 Validation loss: 0.046797 Best loss: 0.046797 Accuracy: 98.63% 9 Validation loss: 0.054517 Best loss: 0.046797 Accuracy: 98.48% 10 Validation loss: 0.046812 Best loss: 0.046797 Accuracy: 98.51% 11 Validation loss: 0.060368 Best loss: 0.046797 Accuracy: 98.48% 12 Validation loss: 0.049366 Best loss: 0.046797 Accuracy: 98.98% 13 Validation loss: 0.082818 Best loss: 0.046797 Accuracy: 98.48% 14 Validation loss: 0.060248 Best loss: 0.046797 Accuracy: 98.05% 15 Validation loss: 0.037658 Best loss: 0.037658 Accuracy: 98.83% 16 Validation loss: 0.085204 Best loss: 0.037658 Accuracy: 98.20% 17 Validation loss: 0.054653 Best loss: 0.037658 Accuracy: 98.79% 18 Validation loss: 0.054907 Best loss: 0.037658 Accuracy: 98.67% 19 Validation loss: 0.050818 Best loss: 0.037658 Accuracy: 98.55% 20 Validation loss: 0.048160 Best loss: 0.037658 Accuracy: 98.67% 21 Validation loss: 0.058002 Best loss: 0.037658 Accuracy: 98.91% 22 Validation loss: 0.049468 Best loss: 0.037658 Accuracy: 98.91% 23 Validation loss: 0.061512 Best loss: 0.037658 Accuracy: 98.98% 24 Validation loss: 0.049333 Best loss: 0.037658 Accuracy: 98.63% 25 Validation loss: 0.054349 Best loss: 0.037658 Accuracy: 98.83% 26 Validation loss: 0.051731 Best loss: 0.037658 Accuracy: 99.06% 27 Validation loss: 0.099475 Best loss: 0.037658 Accuracy: 97.97% 28 Validation loss: 0.088360 Best loss: 0.037658 Accuracy: 98.16% 29 Validation loss: 0.052096 Best loss: 0.037658 Accuracy: 98.83% 30 Validation loss: 0.052343 Best loss: 0.037658 Accuracy: 98.63% 31 Validation loss: 0.114157 Best loss: 0.037658 Accuracy: 98.01% 32 Validation loss: 0.084490 Best loss: 0.037658 Accuracy: 98.59% 33 Validation loss: 0.054856 Best loss: 0.037658 Accuracy: 98.63% 34 Validation loss: 0.071668 Best loss: 0.037658 Accuracy: 98.63% 35 Validation loss: 0.089291 Best loss: 0.037658 Accuracy: 97.89% 36 Validation loss: 0.054619 Best loss: 0.037658 Accuracy: 99.06% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05, total= 6.9min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05 0 Validation loss: 0.134880 Best loss: 0.134880 Accuracy: 96.25% 1 Validation loss: 0.091493 Best loss: 0.091493 Accuracy: 97.69% 2 Validation loss: 0.087084 Best loss: 0.087084 Accuracy: 97.03% 3 Validation loss: 0.066560 Best loss: 0.066560 Accuracy: 98.20% 4 Validation loss: 0.061284 Best loss: 0.061284 Accuracy: 98.32% 5 Validation loss: 0.066194 Best loss: 0.061284 Accuracy: 97.97% 6 Validation loss: 0.237723 Best loss: 0.061284 Accuracy: 94.18% 7 Validation loss: 0.058845 Best loss: 0.058845 Accuracy: 98.40% 8 Validation loss: 0.081291 Best loss: 0.058845 Accuracy: 97.89% 9 Validation loss: 0.063973 Best loss: 0.058845 Accuracy: 98.40% 10 Validation loss: 0.061533 Best loss: 0.058845 Accuracy: 98.36% 11 Validation loss: 0.073591 Best loss: 0.058845 Accuracy: 98.24% 12 Validation loss: 0.052477 Best loss: 0.052477 Accuracy: 98.71% 13 Validation loss: 0.055274 Best loss: 0.052477 Accuracy: 98.16% 14 Validation loss: 0.065075 Best loss: 0.052477 Accuracy: 98.16% 15 Validation loss: 0.095505 Best loss: 0.052477 Accuracy: 97.38% 16 Validation loss: 0.075388 Best loss: 0.052477 Accuracy: 97.93% 17 Validation loss: 0.065356 Best loss: 0.052477 Accuracy: 98.55% 18 Validation loss: 0.046198 Best loss: 0.046198 Accuracy: 98.51% 19 Validation loss: 0.057301 Best loss: 0.046198 Accuracy: 98.32% 20 Validation loss: 0.039556 Best loss: 0.039556 Accuracy: 98.75% 21 Validation loss: 0.041545 Best loss: 0.039556 Accuracy: 98.87% 22 Validation loss: 0.035931 Best loss: 0.035931 Accuracy: 99.30% 23 Validation loss: 0.047301 Best loss: 0.035931 Accuracy: 98.55% 24 Validation loss: 0.046482 Best loss: 0.035931 Accuracy: 98.91% 25 Validation loss: 0.065056 Best loss: 0.035931 Accuracy: 98.51% 26 Validation loss: 0.067309 Best loss: 0.035931 Accuracy: 98.67% 27 Validation loss: 0.167676 Best loss: 0.035931 Accuracy: 96.64% 28 Validation loss: 0.042363 Best loss: 0.035931 Accuracy: 98.98% 29 Validation loss: 0.050610 Best loss: 0.035931 Accuracy: 98.36% 30 Validation loss: 0.052438 Best loss: 0.035931 Accuracy: 98.59% 31 Validation loss: 0.074186 Best loss: 0.035931 Accuracy: 98.05% 32 Validation loss: 0.041782 Best loss: 0.035931 Accuracy: 98.79% 33 Validation loss: 0.043828 Best loss: 0.035931 Accuracy: 98.87% 34 Validation loss: 0.063550 Best loss: 0.035931 Accuracy: 98.63% 35 Validation loss: 0.067869 Best loss: 0.035931 Accuracy: 98.40% 36 Validation loss: 0.052058 Best loss: 0.035931 Accuracy: 99.02% 37 Validation loss: 0.076255 Best loss: 0.035931 Accuracy: 98.24% 38 Validation loss: 0.043133 Best loss: 0.035931 Accuracy: 98.79% 39 Validation loss: 0.062836 Best loss: 0.035931 Accuracy: 98.75% 40 Validation loss: 0.075932 Best loss: 0.035931 Accuracy: 98.44% 41 Validation loss: 0.053302 Best loss: 0.035931 Accuracy: 98.59% 42 Validation loss: 0.088312 Best loss: 0.035931 Accuracy: 98.28% 43 Validation loss: 0.053030 Best loss: 0.035931 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05, total= 8.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05 0 Validation loss: 0.125590 Best loss: 0.125590 Accuracy: 97.19% 1 Validation loss: 0.088656 Best loss: 0.088656 Accuracy: 97.42% 2 Validation loss: 0.072599 Best loss: 0.072599 Accuracy: 97.73% 3 Validation loss: 0.058244 Best loss: 0.058244 Accuracy: 98.16% 4 Validation loss: 0.075648 Best loss: 0.058244 Accuracy: 97.85% 5 Validation loss: 0.065925 Best loss: 0.058244 Accuracy: 97.85% 6 Validation loss: 0.063069 Best loss: 0.058244 Accuracy: 98.24% 7 Validation loss: 0.067473 Best loss: 0.058244 Accuracy: 97.85% 8 Validation loss: 0.050973 Best loss: 0.050973 Accuracy: 98.40% 9 Validation loss: 0.049273 Best loss: 0.049273 Accuracy: 98.40% 10 Validation loss: 0.065048 Best loss: 0.049273 Accuracy: 98.12% 11 Validation loss: 0.060316 Best loss: 0.049273 Accuracy: 98.28% 12 Validation loss: 0.051184 Best loss: 0.049273 Accuracy: 98.48% 13 Validation loss: 0.042958 Best loss: 0.042958 Accuracy: 98.59% 14 Validation loss: 0.069842 Best loss: 0.042958 Accuracy: 97.89% 15 Validation loss: 0.041959 Best loss: 0.041959 Accuracy: 98.75% 16 Validation loss: 0.041549 Best loss: 0.041549 Accuracy: 98.51% 17 Validation loss: 0.036884 Best loss: 0.036884 Accuracy: 98.79% 18 Validation loss: 0.040354 Best loss: 0.036884 Accuracy: 98.98% 19 Validation loss: 0.051883 Best loss: 0.036884 Accuracy: 98.83% 20 Validation loss: 0.045723 Best loss: 0.036884 Accuracy: 98.71% 21 Validation loss: 0.035654 Best loss: 0.035654 Accuracy: 98.94% 22 Validation loss: 0.032752 Best loss: 0.032752 Accuracy: 98.98% 23 Validation loss: 0.041474 Best loss: 0.032752 Accuracy: 98.83% 24 Validation loss: 0.075139 Best loss: 0.032752 Accuracy: 98.28% 25 Validation loss: 0.068763 Best loss: 0.032752 Accuracy: 98.32% 26 Validation loss: 0.048368 Best loss: 0.032752 Accuracy: 98.98% 27 Validation loss: 0.054069 Best loss: 0.032752 Accuracy: 98.16% 28 Validation loss: 0.039736 Best loss: 0.032752 Accuracy: 98.91% 29 Validation loss: 0.057253 Best loss: 0.032752 Accuracy: 98.87% 30 Validation loss: 0.045116 Best loss: 0.032752 Accuracy: 98.83% 31 Validation loss: 0.069160 Best loss: 0.032752 Accuracy: 98.40% 32 Validation loss: 0.032217 Best loss: 0.032217 Accuracy: 99.14% 33 Validation loss: 0.054915 Best loss: 0.032217 Accuracy: 98.79% 34 Validation loss: 0.051533 Best loss: 0.032217 Accuracy: 98.98% 35 Validation loss: 0.073217 Best loss: 0.032217 Accuracy: 98.59% 36 Validation loss: 0.054885 Best loss: 0.032217 Accuracy: 98.59% 37 Validation loss: 0.041721 Best loss: 0.032217 Accuracy: 99.02% 38 Validation loss: 0.047425 Best loss: 0.032217 Accuracy: 98.79% 39 Validation loss: 0.046113 Best loss: 0.032217 Accuracy: 98.75% 40 Validation loss: 0.056096 Best loss: 0.032217 Accuracy: 98.71% 41 Validation loss: 0.053572 Best loss: 0.032217 Accuracy: 98.75% 42 Validation loss: 0.045194 Best loss: 0.032217 Accuracy: 99.02% 43 Validation loss: 0.045951 Best loss: 0.032217 Accuracy: 98.87% 44 Validation loss: 0.045723 Best loss: 0.032217 Accuracy: 98.79% 45 Validation loss: 0.041771 Best loss: 0.032217 Accuracy: 99.18% 46 Validation loss: 0.051239 Best loss: 0.032217 Accuracy: 98.79% 47 Validation loss: 0.058755 Best loss: 0.032217 Accuracy: 98.79% 48 Validation loss: 0.076928 Best loss: 0.032217 Accuracy: 98.28% 49 Validation loss: 0.059109 Best loss: 0.032217 Accuracy: 98.75% 50 Validation loss: 0.054258 Best loss: 0.032217 Accuracy: 99.02% 51 Validation loss: 0.080887 Best loss: 0.032217 Accuracy: 98.79% 52 Validation loss: 0.049741 Best loss: 0.032217 Accuracy: 98.79% 53 Validation loss: 0.054023 Best loss: 0.032217 Accuracy: 98.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.05, total=10.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.115392 Best loss: 0.115392 Accuracy: 97.50% 1 Validation loss: 0.062387 Best loss: 0.062387 Accuracy: 98.44% 2 Validation loss: 0.063104 Best loss: 0.062387 Accuracy: 98.24% 3 Validation loss: 0.061895 Best loss: 0.061895 Accuracy: 98.24% 4 Validation loss: 0.048729 Best loss: 0.048729 Accuracy: 98.67% 5 Validation loss: 0.057426 Best loss: 0.048729 Accuracy: 98.51% 6 Validation loss: 0.033846 Best loss: 0.033846 Accuracy: 98.91% 7 Validation loss: 0.042620 Best loss: 0.033846 Accuracy: 98.83% 8 Validation loss: 0.043474 Best loss: 0.033846 Accuracy: 98.98% 9 Validation loss: 0.088337 Best loss: 0.033846 Accuracy: 97.73% 10 Validation loss: 0.039281 Best loss: 0.033846 Accuracy: 98.94% 11 Validation loss: 0.037959 Best loss: 0.033846 Accuracy: 98.94% 12 Validation loss: 0.068476 Best loss: 0.033846 Accuracy: 98.75% 13 Validation loss: 0.040650 Best loss: 0.033846 Accuracy: 98.91% 14 Validation loss: 0.047443 Best loss: 0.033846 Accuracy: 98.75% 15 Validation loss: 0.056150 Best loss: 0.033846 Accuracy: 99.06% 16 Validation loss: 0.044600 Best loss: 0.033846 Accuracy: 98.79% 17 Validation loss: 0.034741 Best loss: 0.033846 Accuracy: 99.18% 18 Validation loss: 0.053243 Best loss: 0.033846 Accuracy: 98.75% 19 Validation loss: 0.050138 Best loss: 0.033846 Accuracy: 98.83% 20 Validation loss: 0.057900 Best loss: 0.033846 Accuracy: 98.75% 21 Validation loss: 0.055096 Best loss: 0.033846 Accuracy: 98.79% 22 Validation loss: 0.035719 Best loss: 0.033846 Accuracy: 98.91% 23 Validation loss: 0.036796 Best loss: 0.033846 Accuracy: 98.94% 24 Validation loss: 0.042953 Best loss: 0.033846 Accuracy: 98.91% 25 Validation loss: 0.054381 Best loss: 0.033846 Accuracy: 98.98% 26 Validation loss: 0.049837 Best loss: 0.033846 Accuracy: 99.10% 27 Validation loss: 0.060821 Best loss: 0.033846 Accuracy: 98.79% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.106593 Best loss: 0.106593 Accuracy: 97.15% 1 Validation loss: 0.092398 Best loss: 0.092398 Accuracy: 97.54% 2 Validation loss: 0.056506 Best loss: 0.056506 Accuracy: 98.28% 3 Validation loss: 0.040516 Best loss: 0.040516 Accuracy: 98.67% 4 Validation loss: 0.052132 Best loss: 0.040516 Accuracy: 98.48% 5 Validation loss: 0.055699 Best loss: 0.040516 Accuracy: 98.48% 6 Validation loss: 0.034117 Best loss: 0.034117 Accuracy: 98.91% 7 Validation loss: 0.033364 Best loss: 0.033364 Accuracy: 98.87% 8 Validation loss: 0.037504 Best loss: 0.033364 Accuracy: 98.94% 9 Validation loss: 0.064079 Best loss: 0.033364 Accuracy: 98.48% 10 Validation loss: 0.053191 Best loss: 0.033364 Accuracy: 98.83% 11 Validation loss: 0.069395 Best loss: 0.033364 Accuracy: 98.55% 12 Validation loss: 0.052645 Best loss: 0.033364 Accuracy: 98.55% 13 Validation loss: 0.053568 Best loss: 0.033364 Accuracy: 98.63% 14 Validation loss: 0.048648 Best loss: 0.033364 Accuracy: 99.06% 15 Validation loss: 0.068367 Best loss: 0.033364 Accuracy: 98.71% 16 Validation loss: 0.042430 Best loss: 0.033364 Accuracy: 99.06% 17 Validation loss: 0.040851 Best loss: 0.033364 Accuracy: 99.10% 18 Validation loss: 0.049638 Best loss: 0.033364 Accuracy: 98.91% 19 Validation loss: 0.051085 Best loss: 0.033364 Accuracy: 98.87% 20 Validation loss: 0.046789 Best loss: 0.033364 Accuracy: 99.06% 21 Validation loss: 0.049773 Best loss: 0.033364 Accuracy: 98.83% 22 Validation loss: 0.063842 Best loss: 0.033364 Accuracy: 98.83% 23 Validation loss: 0.039500 Best loss: 0.033364 Accuracy: 98.91% 24 Validation loss: 0.057690 Best loss: 0.033364 Accuracy: 98.83% 25 Validation loss: 0.046828 Best loss: 0.033364 Accuracy: 99.14% 26 Validation loss: 0.053360 Best loss: 0.033364 Accuracy: 98.87% 27 Validation loss: 0.035936 Best loss: 0.033364 Accuracy: 99.18% 28 Validation loss: 0.043861 Best loss: 0.033364 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.115637 Best loss: 0.115637 Accuracy: 97.34% 1 Validation loss: 0.051563 Best loss: 0.051563 Accuracy: 98.40% 2 Validation loss: 0.075184 Best loss: 0.051563 Accuracy: 97.73% 3 Validation loss: 0.042135 Best loss: 0.042135 Accuracy: 98.63% 4 Validation loss: 0.038906 Best loss: 0.038906 Accuracy: 98.87% 5 Validation loss: 0.075887 Best loss: 0.038906 Accuracy: 98.40% 6 Validation loss: 0.051270 Best loss: 0.038906 Accuracy: 98.59% 7 Validation loss: 0.043257 Best loss: 0.038906 Accuracy: 98.79% 8 Validation loss: 0.046081 Best loss: 0.038906 Accuracy: 98.83% 9 Validation loss: 0.041766 Best loss: 0.038906 Accuracy: 98.83% 10 Validation loss: 0.045753 Best loss: 0.038906 Accuracy: 98.79% 11 Validation loss: 0.036824 Best loss: 0.036824 Accuracy: 98.75% 12 Validation loss: 0.037367 Best loss: 0.036824 Accuracy: 98.87% 13 Validation loss: 0.034370 Best loss: 0.034370 Accuracy: 99.06% 14 Validation loss: 0.051501 Best loss: 0.034370 Accuracy: 98.71% 15 Validation loss: 0.040759 Best loss: 0.034370 Accuracy: 98.94% 16 Validation loss: 0.045550 Best loss: 0.034370 Accuracy: 99.02% 17 Validation loss: 0.035947 Best loss: 0.034370 Accuracy: 99.14% 18 Validation loss: 0.044423 Best loss: 0.034370 Accuracy: 99.22% 19 Validation loss: 0.038549 Best loss: 0.034370 Accuracy: 99.22% 20 Validation loss: 0.055510 Best loss: 0.034370 Accuracy: 98.51% 21 Validation loss: 0.041878 Best loss: 0.034370 Accuracy: 98.98% 22 Validation loss: 0.048909 Best loss: 0.034370 Accuracy: 98.83% 23 Validation loss: 0.059799 Best loss: 0.034370 Accuracy: 98.63% 24 Validation loss: 0.035444 Best loss: 0.034370 Accuracy: 99.18% 25 Validation loss: 0.025220 Best loss: 0.025220 Accuracy: 99.34% 26 Validation loss: 0.041332 Best loss: 0.025220 Accuracy: 99.22% 27 Validation loss: 0.034641 Best loss: 0.025220 Accuracy: 99.34% 28 Validation loss: 0.046491 Best loss: 0.025220 Accuracy: 98.98% 29 Validation loss: 0.054095 Best loss: 0.025220 Accuracy: 98.79% 30 Validation loss: 0.040454 Best loss: 0.025220 Accuracy: 99.14% 31 Validation loss: 0.052271 Best loss: 0.025220 Accuracy: 98.98% 32 Validation loss: 0.045470 Best loss: 0.025220 Accuracy: 99.10% 33 Validation loss: 0.035160 Best loss: 0.025220 Accuracy: 99.45% 34 Validation loss: 0.034749 Best loss: 0.025220 Accuracy: 99.22% 35 Validation loss: 0.038297 Best loss: 0.025220 Accuracy: 99.26% 36 Validation loss: 0.041696 Best loss: 0.025220 Accuracy: 99.26% 37 Validation loss: 0.055726 Best loss: 0.025220 Accuracy: 99.02% 38 Validation loss: 0.047098 Best loss: 0.025220 Accuracy: 99.14% 39 Validation loss: 0.077571 Best loss: 0.025220 Accuracy: 98.08% 40 Validation loss: 0.045528 Best loss: 0.025220 Accuracy: 99.26% 41 Validation loss: 0.044766 Best loss: 0.025220 Accuracy: 99.22% 42 Validation loss: 0.042319 Best loss: 0.025220 Accuracy: 99.14% 43 Validation loss: 0.035859 Best loss: 0.025220 Accuracy: 99.18% 44 Validation loss: 0.044023 Best loss: 0.025220 Accuracy: 99.22% 45 Validation loss: 0.046162 Best loss: 0.025220 Accuracy: 99.06% 46 Validation loss: 0.049399 Best loss: 0.025220 Accuracy: 99.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=120, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.01, total= 2.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 496.794708 Best loss: 496.794708 Accuracy: 77.72% 1 Validation loss: 47.979530 Best loss: 47.979530 Accuracy: 84.52% 2 Validation loss: 8.472088 Best loss: 8.472088 Accuracy: 92.10% 3 Validation loss: 3.235445 Best loss: 3.235445 Accuracy: 94.57% 4 Validation loss: 3.137283 Best loss: 3.137283 Accuracy: 93.63% 5 Validation loss: 0.933185 Best loss: 0.933185 Accuracy: 97.34% 6 Validation loss: 0.881580 Best loss: 0.881580 Accuracy: 96.72% 7 Validation loss: 1.641470 Best loss: 0.881580 Accuracy: 95.11% 8 Validation loss: 0.598984 Best loss: 0.598984 Accuracy: 96.99% 9 Validation loss: 0.684902 Best loss: 0.598984 Accuracy: 97.26% 10 Validation loss: 0.619601 Best loss: 0.598984 Accuracy: 97.81% 11 Validation loss: 0.647796 Best loss: 0.598984 Accuracy: 97.42% 12 Validation loss: 0.675678 Best loss: 0.598984 Accuracy: 97.38% 13 Validation loss: 0.823696 Best loss: 0.598984 Accuracy: 97.03% 14 Validation loss: 0.531173 Best loss: 0.531173 Accuracy: 97.46% 15 Validation loss: 0.469348 Best loss: 0.469348 Accuracy: 97.77% 16 Validation loss: 0.424620 Best loss: 0.424620 Accuracy: 97.69% 17 Validation loss: 0.416381 Best loss: 0.416381 Accuracy: 97.93% 18 Validation loss: 0.410402 Best loss: 0.410402 Accuracy: 97.81% 19 Validation loss: 0.697412 Best loss: 0.410402 Accuracy: 97.03% 20 Validation loss: 0.406598 Best loss: 0.406598 Accuracy: 98.16% 21 Validation loss: 0.410932 Best loss: 0.406598 Accuracy: 97.85% 22 Validation loss: 0.404606 Best loss: 0.404606 Accuracy: 98.12% 23 Validation loss: 0.665423 Best loss: 0.404606 Accuracy: 96.56% 24 Validation loss: 0.319486 Best loss: 0.319486 Accuracy: 98.36% 25 Validation loss: 0.324077 Best loss: 0.319486 Accuracy: 98.20% 26 Validation loss: 0.335896 Best loss: 0.319486 Accuracy: 98.01% 27 Validation loss: 0.284293 Best loss: 0.284293 Accuracy: 98.24% 28 Validation loss: 0.326728 Best loss: 0.284293 Accuracy: 98.28% 29 Validation loss: 0.456860 Best loss: 0.284293 Accuracy: 97.38% 30 Validation loss: 0.246580 Best loss: 0.246580 Accuracy: 98.63% 31 Validation loss: 0.243784 Best loss: 0.243784 Accuracy: 98.79% 32 Validation loss: 0.254966 Best loss: 0.243784 Accuracy: 98.12% 33 Validation loss: 0.224955 Best loss: 0.224955 Accuracy: 98.67% 34 Validation loss: 0.356885 Best loss: 0.224955 Accuracy: 98.05% 35 Validation loss: 0.432450 Best loss: 0.224955 Accuracy: 97.50% 36 Validation loss: 0.242790 Best loss: 0.224955 Accuracy: 98.16% 37 Validation loss: 0.250640 Best loss: 0.224955 Accuracy: 98.20% 38 Validation loss: 0.231848 Best loss: 0.224955 Accuracy: 98.48% 39 Validation loss: 0.248181 Best loss: 0.224955 Accuracy: 98.24% 40 Validation loss: 0.244288 Best loss: 0.224955 Accuracy: 98.44% 41 Validation loss: 0.223673 Best loss: 0.223673 Accuracy: 98.40% 42 Validation loss: 0.204167 Best loss: 0.204167 Accuracy: 98.83% 43 Validation loss: 0.213674 Best loss: 0.204167 Accuracy: 98.71% 44 Validation loss: 0.264814 Best loss: 0.204167 Accuracy: 98.40% 45 Validation loss: 0.199643 Best loss: 0.199643 Accuracy: 98.55% 46 Validation loss: 0.233414 Best loss: 0.199643 Accuracy: 98.40% 47 Validation loss: 0.265203 Best loss: 0.199643 Accuracy: 97.73% 48 Validation loss: 0.256023 Best loss: 0.199643 Accuracy: 98.20% 49 Validation loss: 0.218348 Best loss: 0.199643 Accuracy: 98.48% 50 Validation loss: 0.227558 Best loss: 0.199643 Accuracy: 98.12% 51 Validation loss: 0.165364 Best loss: 0.165364 Accuracy: 98.75% 52 Validation loss: 0.159500 Best loss: 0.159500 Accuracy: 98.63% 53 Validation loss: 0.169609 Best loss: 0.159500 Accuracy: 98.75% 54 Validation loss: 0.209587 Best loss: 0.159500 Accuracy: 98.28% 55 Validation loss: 0.281353 Best loss: 0.159500 Accuracy: 97.65% 56 Validation loss: 0.205979 Best loss: 0.159500 Accuracy: 98.51% 57 Validation loss: 0.244357 Best loss: 0.159500 Accuracy: 98.32% 58 Validation loss: 0.179361 Best loss: 0.159500 Accuracy: 98.87% 59 Validation loss: 0.193499 Best loss: 0.159500 Accuracy: 98.75% 60 Validation loss: 0.239696 Best loss: 0.159500 Accuracy: 98.59% 61 Validation loss: 0.216734 Best loss: 0.159500 Accuracy: 98.32% 62 Validation loss: 0.234239 Best loss: 0.159500 Accuracy: 98.71% 63 Validation loss: 0.374141 Best loss: 0.159500 Accuracy: 97.65% 64 Validation loss: 0.229566 Best loss: 0.159500 Accuracy: 98.44% 65 Validation loss: 0.209958 Best loss: 0.159500 Accuracy: 98.55% 66 Validation loss: 0.206844 Best loss: 0.159500 Accuracy: 98.79% 67 Validation loss: 0.231299 Best loss: 0.159500 Accuracy: 98.55% 68 Validation loss: 0.195088 Best loss: 0.159500 Accuracy: 98.75% 69 Validation loss: 0.184879 Best loss: 0.159500 Accuracy: 98.91% 70 Validation loss: 0.172057 Best loss: 0.159500 Accuracy: 98.91% 71 Validation loss: 0.160800 Best loss: 0.159500 Accuracy: 98.87% 72 Validation loss: 0.162360 Best loss: 0.159500 Accuracy: 98.83% 73 Validation loss: 0.153759 Best loss: 0.153759 Accuracy: 98.83% 74 Validation loss: 0.148118 Best loss: 0.148118 Accuracy: 98.87% 75 Validation loss: 0.142924 Best loss: 0.142924 Accuracy: 98.91% 76 Validation loss: 0.136942 Best loss: 0.136942 Accuracy: 98.91% 77 Validation loss: 0.133369 Best loss: 0.133369 Accuracy: 98.91% 78 Validation loss: 0.138085 Best loss: 0.133369 Accuracy: 98.91% 79 Validation loss: 0.119819 Best loss: 0.119819 Accuracy: 99.02% 80 Validation loss: 0.123523 Best loss: 0.119819 Accuracy: 98.94% 81 Validation loss: 0.120099 Best loss: 0.119819 Accuracy: 98.94% 82 Validation loss: 0.117540 Best loss: 0.117540 Accuracy: 98.91% 83 Validation loss: 0.115234 Best loss: 0.115234 Accuracy: 98.94% 84 Validation loss: 0.109503 Best loss: 0.109503 Accuracy: 98.91% 85 Validation loss: 0.106848 Best loss: 0.106848 Accuracy: 98.94% 86 Validation loss: 0.105426 Best loss: 0.105426 Accuracy: 98.94% 87 Validation loss: 0.102904 Best loss: 0.102904 Accuracy: 98.94% 88 Validation loss: 0.103801 Best loss: 0.102904 Accuracy: 98.98% 89 Validation loss: 0.103023 Best loss: 0.102904 Accuracy: 98.98% 90 Validation loss: 0.101530 Best loss: 0.101530 Accuracy: 98.98% 91 Validation loss: 0.100125 Best loss: 0.100125 Accuracy: 98.98% 92 Validation loss: 0.100458 Best loss: 0.100125 Accuracy: 99.02% 93 Validation loss: 0.099460 Best loss: 0.099460 Accuracy: 99.02% 94 Validation loss: 0.098105 Best loss: 0.098105 Accuracy: 99.02% 95 Validation loss: 0.095604 Best loss: 0.095604 Accuracy: 98.98% 96 Validation loss: 0.094847 Best loss: 0.094847 Accuracy: 99.02% 97 Validation loss: 0.094199 Best loss: 0.094199 Accuracy: 99.02% 98 Validation loss: 0.093173 Best loss: 0.093173 Accuracy: 98.98% 99 Validation loss: 0.092265 Best loss: 0.092265 Accuracy: 98.98% 100 Validation loss: 0.091630 Best loss: 0.091630 Accuracy: 98.98% 101 Validation loss: 0.090821 Best loss: 0.090821 Accuracy: 98.98% 102 Validation loss: 0.090127 Best loss: 0.090127 Accuracy: 98.98% 103 Validation loss: 0.089455 Best loss: 0.089455 Accuracy: 98.98% 104 Validation loss: 0.088842 Best loss: 0.088842 Accuracy: 99.02% 105 Validation loss: 0.088268 Best loss: 0.088268 Accuracy: 99.02% 106 Validation loss: 0.087659 Best loss: 0.087659 Accuracy: 99.02% 107 Validation loss: 0.087080 Best loss: 0.087080 Accuracy: 99.02% 108 Validation loss: 0.086510 Best loss: 0.086510 Accuracy: 99.06% 109 Validation loss: 0.086512 Best loss: 0.086510 Accuracy: 99.06% 110 Validation loss: 0.086132 Best loss: 0.086132 Accuracy: 99.02% 111 Validation loss: 0.085816 Best loss: 0.085816 Accuracy: 99.02% 112 Validation loss: 0.085563 Best loss: 0.085563 Accuracy: 99.02% 113 Validation loss: 0.085300 Best loss: 0.085300 Accuracy: 99.02% 114 Validation loss: 0.085354 Best loss: 0.085300 Accuracy: 99.02% 115 Validation loss: 0.084992 Best loss: 0.084992 Accuracy: 99.06% 116 Validation loss: 0.084218 Best loss: 0.084218 Accuracy: 99.10% 117 Validation loss: 0.083567 Best loss: 0.083567 Accuracy: 99.10% 118 Validation loss: 0.083191 Best loss: 0.083191 Accuracy: 99.10% 119 Validation loss: 0.082766 Best loss: 0.082766 Accuracy: 99.10% 120 Validation loss: 0.082482 Best loss: 0.082482 Accuracy: 99.10% 121 Validation loss: 0.082164 Best loss: 0.082164 Accuracy: 99.10% 122 Validation loss: 0.081840 Best loss: 0.081840 Accuracy: 99.10% 123 Validation loss: 0.081674 Best loss: 0.081674 Accuracy: 99.02% 124 Validation loss: 0.081913 Best loss: 0.081674 Accuracy: 99.02% 125 Validation loss: 0.080877 Best loss: 0.080877 Accuracy: 99.02% 126 Validation loss: 0.079944 Best loss: 0.079944 Accuracy: 99.02% 127 Validation loss: 0.090917 Best loss: 0.079944 Accuracy: 98.98% 128 Validation loss: 3.898789 Best loss: 0.079944 Accuracy: 93.67% 129 Validation loss: 2.581474 Best loss: 0.079944 Accuracy: 98.24% 130 Validation loss: 3.160921 Best loss: 0.079944 Accuracy: 97.77% 131 Validation loss: 1.696300 Best loss: 0.079944 Accuracy: 98.08% 132 Validation loss: 1.398082 Best loss: 0.079944 Accuracy: 97.89% 133 Validation loss: 0.893439 Best loss: 0.079944 Accuracy: 98.71% 134 Validation loss: 0.751248 Best loss: 0.079944 Accuracy: 98.63% 135 Validation loss: 0.684702 Best loss: 0.079944 Accuracy: 98.79% 136 Validation loss: 0.518062 Best loss: 0.079944 Accuracy: 98.98% 137 Validation loss: 0.442312 Best loss: 0.079944 Accuracy: 98.94% 138 Validation loss: 0.381464 Best loss: 0.079944 Accuracy: 99.02% 139 Validation loss: 0.356939 Best loss: 0.079944 Accuracy: 99.18% 140 Validation loss: 0.313590 Best loss: 0.079944 Accuracy: 99.06% 141 Validation loss: 0.364054 Best loss: 0.079944 Accuracy: 98.91% 142 Validation loss: 0.345462 Best loss: 0.079944 Accuracy: 98.79% 143 Validation loss: 0.280714 Best loss: 0.079944 Accuracy: 98.71% 144 Validation loss: 0.229679 Best loss: 0.079944 Accuracy: 98.91% 145 Validation loss: 0.235101 Best loss: 0.079944 Accuracy: 98.98% 146 Validation loss: 0.286424 Best loss: 0.079944 Accuracy: 98.83% 147 Validation loss: 0.235349 Best loss: 0.079944 Accuracy: 98.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05, total= 45.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 98.025627 Best loss: 98.025627 Accuracy: 91.79% 1 Validation loss: 20.001778 Best loss: 20.001778 Accuracy: 93.67% 2 Validation loss: 6.123318 Best loss: 6.123318 Accuracy: 95.58% 3 Validation loss: 4.960836 Best loss: 4.960836 Accuracy: 94.64% 4 Validation loss: 2.780573 Best loss: 2.780573 Accuracy: 96.05% 5 Validation loss: 1.922089 Best loss: 1.922089 Accuracy: 96.99% 6 Validation loss: 1.019517 Best loss: 1.019517 Accuracy: 97.73% 7 Validation loss: 1.231643 Best loss: 1.019517 Accuracy: 97.03% 8 Validation loss: 0.923499 Best loss: 0.923499 Accuracy: 97.22% 9 Validation loss: 0.732103 Best loss: 0.732103 Accuracy: 97.58% 10 Validation loss: 0.514090 Best loss: 0.514090 Accuracy: 98.40% 11 Validation loss: 0.924081 Best loss: 0.514090 Accuracy: 97.11% 12 Validation loss: 0.729147 Best loss: 0.514090 Accuracy: 97.73% 13 Validation loss: 0.731045 Best loss: 0.514090 Accuracy: 97.62% 14 Validation loss: 0.783325 Best loss: 0.514090 Accuracy: 97.69% 15 Validation loss: 1.526488 Best loss: 0.514090 Accuracy: 94.64% 16 Validation loss: 0.697330 Best loss: 0.514090 Accuracy: 97.50% 17 Validation loss: 0.531859 Best loss: 0.514090 Accuracy: 98.01% 18 Validation loss: 0.586521 Best loss: 0.514090 Accuracy: 97.81% 19 Validation loss: 1.323289 Best loss: 0.514090 Accuracy: 94.96% 20 Validation loss: 0.400404 Best loss: 0.400404 Accuracy: 98.12% 21 Validation loss: 0.680480 Best loss: 0.400404 Accuracy: 96.79% 22 Validation loss: 0.616651 Best loss: 0.400404 Accuracy: 97.85% 23 Validation loss: 0.580719 Best loss: 0.400404 Accuracy: 97.77% 24 Validation loss: 0.616040 Best loss: 0.400404 Accuracy: 97.34% 25 Validation loss: 0.608198 Best loss: 0.400404 Accuracy: 96.87% 26 Validation loss: 0.448623 Best loss: 0.400404 Accuracy: 98.36% 27 Validation loss: 0.406093 Best loss: 0.400404 Accuracy: 98.44% 28 Validation loss: 0.460439 Best loss: 0.400404 Accuracy: 98.16% 29 Validation loss: 0.324763 Best loss: 0.324763 Accuracy: 98.48% 30 Validation loss: 0.356759 Best loss: 0.324763 Accuracy: 98.28% 31 Validation loss: 0.245812 Best loss: 0.245812 Accuracy: 98.67% 32 Validation loss: 0.252267 Best loss: 0.245812 Accuracy: 98.59% 33 Validation loss: 0.258651 Best loss: 0.245812 Accuracy: 98.71% 34 Validation loss: 0.246263 Best loss: 0.245812 Accuracy: 98.83% 35 Validation loss: 0.226863 Best loss: 0.226863 Accuracy: 98.75% 36 Validation loss: 0.208973 Best loss: 0.208973 Accuracy: 98.83% 37 Validation loss: 0.198430 Best loss: 0.198430 Accuracy: 98.71% 38 Validation loss: 0.178262 Best loss: 0.178262 Accuracy: 98.67% 39 Validation loss: 0.310982 Best loss: 0.178262 Accuracy: 98.20% 40 Validation loss: 0.337073 Best loss: 0.178262 Accuracy: 97.62% 41 Validation loss: 0.326689 Best loss: 0.178262 Accuracy: 97.69% 42 Validation loss: 0.301517 Best loss: 0.178262 Accuracy: 98.12% 43 Validation loss: 0.284781 Best loss: 0.178262 Accuracy: 98.08% 44 Validation loss: 0.306931 Best loss: 0.178262 Accuracy: 97.97% 45 Validation loss: 0.228130 Best loss: 0.178262 Accuracy: 98.51% 46 Validation loss: 0.276320 Best loss: 0.178262 Accuracy: 98.32% 47 Validation loss: 0.304680 Best loss: 0.178262 Accuracy: 97.85% 48 Validation loss: 0.347904 Best loss: 0.178262 Accuracy: 97.81% 49 Validation loss: 0.226015 Best loss: 0.178262 Accuracy: 98.51% 50 Validation loss: 0.274883 Best loss: 0.178262 Accuracy: 98.20% 51 Validation loss: 0.431165 Best loss: 0.178262 Accuracy: 97.69% 52 Validation loss: 0.280670 Best loss: 0.178262 Accuracy: 98.36% 53 Validation loss: 0.337788 Best loss: 0.178262 Accuracy: 98.08% 54 Validation loss: 0.328510 Best loss: 0.178262 Accuracy: 97.50% 55 Validation loss: 0.181117 Best loss: 0.178262 Accuracy: 98.71% 56 Validation loss: 0.199194 Best loss: 0.178262 Accuracy: 98.48% 57 Validation loss: 0.300698 Best loss: 0.178262 Accuracy: 98.32% 58 Validation loss: 0.534171 Best loss: 0.178262 Accuracy: 96.36% 59 Validation loss: 0.252936 Best loss: 0.178262 Accuracy: 98.63% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05, total= 20.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 532.184631 Best loss: 532.184631 Accuracy: 80.26% 1 Validation loss: 35.187836 Best loss: 35.187836 Accuracy: 88.90% 2 Validation loss: 13.202633 Best loss: 13.202633 Accuracy: 89.56% 3 Validation loss: 7.054620 Best loss: 7.054620 Accuracy: 89.33% 4 Validation loss: 5.531249 Best loss: 5.531249 Accuracy: 90.66% 5 Validation loss: 1.187631 Best loss: 1.187631 Accuracy: 97.38% 6 Validation loss: 2.092070 Best loss: 1.187631 Accuracy: 95.35% 7 Validation loss: 1.479785 Best loss: 1.187631 Accuracy: 94.72% 8 Validation loss: 0.958759 Best loss: 0.958759 Accuracy: 96.56% 9 Validation loss: 0.627357 Best loss: 0.627357 Accuracy: 98.05% 10 Validation loss: 0.511498 Best loss: 0.511498 Accuracy: 97.93% 11 Validation loss: 0.426546 Best loss: 0.426546 Accuracy: 98.44% 12 Validation loss: 1.054756 Best loss: 0.426546 Accuracy: 95.47% 13 Validation loss: 0.977744 Best loss: 0.426546 Accuracy: 95.82% 14 Validation loss: 0.499832 Best loss: 0.426546 Accuracy: 97.30% 15 Validation loss: 0.394334 Best loss: 0.394334 Accuracy: 98.24% 16 Validation loss: 0.529490 Best loss: 0.394334 Accuracy: 97.97% 17 Validation loss: 0.538159 Best loss: 0.394334 Accuracy: 97.46% 18 Validation loss: 0.533197 Best loss: 0.394334 Accuracy: 97.73% 19 Validation loss: 0.452097 Best loss: 0.394334 Accuracy: 97.97% 20 Validation loss: 0.871064 Best loss: 0.394334 Accuracy: 95.54% 21 Validation loss: 0.474339 Best loss: 0.394334 Accuracy: 97.46% 22 Validation loss: 0.327851 Best loss: 0.327851 Accuracy: 98.16% 23 Validation loss: 0.767832 Best loss: 0.327851 Accuracy: 96.05% 24 Validation loss: 0.462837 Best loss: 0.327851 Accuracy: 97.93% 25 Validation loss: 0.425026 Best loss: 0.327851 Accuracy: 97.77% 26 Validation loss: 0.383870 Best loss: 0.327851 Accuracy: 98.08% 27 Validation loss: 0.413243 Best loss: 0.327851 Accuracy: 97.54% 28 Validation loss: 0.320878 Best loss: 0.320878 Accuracy: 98.12% 29 Validation loss: 0.314155 Best loss: 0.314155 Accuracy: 98.24% 30 Validation loss: 0.332332 Best loss: 0.314155 Accuracy: 98.36% 31 Validation loss: 0.245447 Best loss: 0.245447 Accuracy: 98.51% 32 Validation loss: 0.251608 Best loss: 0.245447 Accuracy: 98.40% 33 Validation loss: 0.223106 Best loss: 0.223106 Accuracy: 98.55% 34 Validation loss: 0.235780 Best loss: 0.223106 Accuracy: 98.48% 35 Validation loss: 0.247954 Best loss: 0.223106 Accuracy: 98.28% 36 Validation loss: 0.185760 Best loss: 0.185760 Accuracy: 98.75% 37 Validation loss: 0.168462 Best loss: 0.168462 Accuracy: 99.02% 38 Validation loss: 0.164062 Best loss: 0.164062 Accuracy: 99.02% 39 Validation loss: 0.152534 Best loss: 0.152534 Accuracy: 98.98% 40 Validation loss: 0.162969 Best loss: 0.152534 Accuracy: 98.55% 41 Validation loss: 0.217445 Best loss: 0.152534 Accuracy: 98.44% 42 Validation loss: 0.260442 Best loss: 0.152534 Accuracy: 97.97% 43 Validation loss: 0.189937 Best loss: 0.152534 Accuracy: 98.67% 44 Validation loss: 0.207292 Best loss: 0.152534 Accuracy: 98.12% 45 Validation loss: 0.246418 Best loss: 0.152534 Accuracy: 98.63% 46 Validation loss: 0.235096 Best loss: 0.152534 Accuracy: 98.16% 47 Validation loss: 0.154447 Best loss: 0.152534 Accuracy: 98.98% 48 Validation loss: 0.279329 Best loss: 0.152534 Accuracy: 98.24% 49 Validation loss: 0.286551 Best loss: 0.152534 Accuracy: 98.20% 50 Validation loss: 0.191301 Best loss: 0.152534 Accuracy: 98.79% 51 Validation loss: 0.279971 Best loss: 0.152534 Accuracy: 98.32% 52 Validation loss: 0.175022 Best loss: 0.152534 Accuracy: 98.91% 53 Validation loss: 0.168847 Best loss: 0.152534 Accuracy: 98.87% 54 Validation loss: 0.211363 Best loss: 0.152534 Accuracy: 98.55% 55 Validation loss: 0.231330 Best loss: 0.152534 Accuracy: 98.63% 56 Validation loss: 0.252426 Best loss: 0.152534 Accuracy: 98.16% 57 Validation loss: 0.257774 Best loss: 0.152534 Accuracy: 98.51% 58 Validation loss: 0.146677 Best loss: 0.146677 Accuracy: 98.87% 59 Validation loss: 0.182510 Best loss: 0.146677 Accuracy: 98.94% 60 Validation loss: 0.225821 Best loss: 0.146677 Accuracy: 98.63% 61 Validation loss: 0.212359 Best loss: 0.146677 Accuracy: 98.51% 62 Validation loss: 0.203375 Best loss: 0.146677 Accuracy: 98.51% 63 Validation loss: 0.234726 Best loss: 0.146677 Accuracy: 98.71% 64 Validation loss: 0.147265 Best loss: 0.146677 Accuracy: 98.91% 65 Validation loss: 0.226999 Best loss: 0.146677 Accuracy: 98.79% 66 Validation loss: 0.207206 Best loss: 0.146677 Accuracy: 98.71% 67 Validation loss: 0.174419 Best loss: 0.146677 Accuracy: 98.67% 68 Validation loss: 0.200004 Best loss: 0.146677 Accuracy: 98.40% 69 Validation loss: 0.179540 Best loss: 0.146677 Accuracy: 98.79% 70 Validation loss: 0.186071 Best loss: 0.146677 Accuracy: 98.91% 71 Validation loss: 0.170322 Best loss: 0.146677 Accuracy: 98.87% 72 Validation loss: 0.163727 Best loss: 0.146677 Accuracy: 99.10% 73 Validation loss: 0.157460 Best loss: 0.146677 Accuracy: 99.10% 74 Validation loss: 0.150104 Best loss: 0.146677 Accuracy: 99.10% 75 Validation loss: 0.142238 Best loss: 0.142238 Accuracy: 99.14% 76 Validation loss: 0.131368 Best loss: 0.131368 Accuracy: 99.14% 77 Validation loss: 0.131998 Best loss: 0.131368 Accuracy: 99.18% 78 Validation loss: 0.129805 Best loss: 0.129805 Accuracy: 99.14% 79 Validation loss: 0.121114 Best loss: 0.121114 Accuracy: 99.06% 80 Validation loss: 0.117485 Best loss: 0.117485 Accuracy: 99.06% 81 Validation loss: 0.113735 Best loss: 0.113735 Accuracy: 98.98% 82 Validation loss: 0.111051 Best loss: 0.111051 Accuracy: 99.02% 83 Validation loss: 0.107519 Best loss: 0.107519 Accuracy: 99.02% 84 Validation loss: 0.105035 Best loss: 0.105035 Accuracy: 99.02% 85 Validation loss: 0.102592 Best loss: 0.102592 Accuracy: 99.06% 86 Validation loss: 0.099613 Best loss: 0.099613 Accuracy: 99.02% 87 Validation loss: 0.096369 Best loss: 0.096369 Accuracy: 99.06% 88 Validation loss: 0.095240 Best loss: 0.095240 Accuracy: 99.10% 89 Validation loss: 0.093143 Best loss: 0.093143 Accuracy: 99.10% 90 Validation loss: 0.091469 Best loss: 0.091469 Accuracy: 99.06% 91 Validation loss: 0.089613 Best loss: 0.089613 Accuracy: 99.06% 92 Validation loss: 0.098907 Best loss: 0.089613 Accuracy: 98.94% 93 Validation loss: 0.094445 Best loss: 0.089613 Accuracy: 99.06% 94 Validation loss: 0.093897 Best loss: 0.089613 Accuracy: 98.98% 95 Validation loss: 0.119051 Best loss: 0.089613 Accuracy: 98.87% 96 Validation loss: 0.164904 Best loss: 0.089613 Accuracy: 98.16% 97 Validation loss: 0.110757 Best loss: 0.089613 Accuracy: 98.63% 98 Validation loss: 0.293637 Best loss: 0.089613 Accuracy: 98.20% 99 Validation loss: 0.317864 Best loss: 0.089613 Accuracy: 98.55% 100 Validation loss: 0.358879 Best loss: 0.089613 Accuracy: 98.51% 101 Validation loss: 0.270627 Best loss: 0.089613 Accuracy: 98.63% 102 Validation loss: 0.232658 Best loss: 0.089613 Accuracy: 98.98% 103 Validation loss: 0.415392 Best loss: 0.089613 Accuracy: 98.20% 104 Validation loss: 0.485725 Best loss: 0.089613 Accuracy: 97.77% 105 Validation loss: 0.427084 Best loss: 0.089613 Accuracy: 98.48% 106 Validation loss: 0.290189 Best loss: 0.089613 Accuracy: 98.67% 107 Validation loss: 0.422101 Best loss: 0.089613 Accuracy: 98.36% 108 Validation loss: 0.308356 Best loss: 0.089613 Accuracy: 98.83% 109 Validation loss: 0.273258 Best loss: 0.089613 Accuracy: 98.94% 110 Validation loss: 0.319805 Best loss: 0.089613 Accuracy: 98.87% 111 Validation loss: 0.310851 Best loss: 0.089613 Accuracy: 98.79% 112 Validation loss: 0.280825 Best loss: 0.089613 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.05, total= 35.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.080938 Best loss: 0.080938 Accuracy: 97.85% 1 Validation loss: 0.058630 Best loss: 0.058630 Accuracy: 98.24% 2 Validation loss: 0.053123 Best loss: 0.053123 Accuracy: 98.44% 3 Validation loss: 0.049720 Best loss: 0.049720 Accuracy: 98.48% 4 Validation loss: 0.047576 Best loss: 0.047576 Accuracy: 98.63% 5 Validation loss: 0.068058 Best loss: 0.047576 Accuracy: 98.36% 6 Validation loss: 0.071141 Best loss: 0.047576 Accuracy: 98.36% 7 Validation loss: 0.051346 Best loss: 0.047576 Accuracy: 98.63% 8 Validation loss: 0.049628 Best loss: 0.047576 Accuracy: 98.67% 9 Validation loss: 0.038319 Best loss: 0.038319 Accuracy: 99.10% 10 Validation loss: 0.039908 Best loss: 0.038319 Accuracy: 99.06% 11 Validation loss: 0.046465 Best loss: 0.038319 Accuracy: 98.98% 12 Validation loss: 0.047356 Best loss: 0.038319 Accuracy: 99.06% 13 Validation loss: 0.052969 Best loss: 0.038319 Accuracy: 98.67% 14 Validation loss: 0.056806 Best loss: 0.038319 Accuracy: 98.59% 15 Validation loss: 0.059641 Best loss: 0.038319 Accuracy: 98.83% 16 Validation loss: 0.050392 Best loss: 0.038319 Accuracy: 98.79% 17 Validation loss: 0.062437 Best loss: 0.038319 Accuracy: 98.79% 18 Validation loss: 0.054753 Best loss: 0.038319 Accuracy: 98.83% 19 Validation loss: 0.050183 Best loss: 0.038319 Accuracy: 98.79% 20 Validation loss: 0.038607 Best loss: 0.038319 Accuracy: 99.06% 21 Validation loss: 0.050555 Best loss: 0.038319 Accuracy: 98.91% 22 Validation loss: 0.040587 Best loss: 0.038319 Accuracy: 99.06% 23 Validation loss: 0.038951 Best loss: 0.038319 Accuracy: 99.18% 24 Validation loss: 0.034177 Best loss: 0.034177 Accuracy: 99.14% 25 Validation loss: 0.035804 Best loss: 0.034177 Accuracy: 99.22% 26 Validation loss: 0.037562 Best loss: 0.034177 Accuracy: 99.22% 27 Validation loss: 0.031563 Best loss: 0.031563 Accuracy: 99.41% 28 Validation loss: 0.033817 Best loss: 0.031563 Accuracy: 99.45% 29 Validation loss: 0.032403 Best loss: 0.031563 Accuracy: 99.45% 30 Validation loss: 0.040540 Best loss: 0.031563 Accuracy: 99.34% 31 Validation loss: 0.045616 Best loss: 0.031563 Accuracy: 99.22% 32 Validation loss: 0.064422 Best loss: 0.031563 Accuracy: 98.79% 33 Validation loss: 0.058267 Best loss: 0.031563 Accuracy: 98.91% 34 Validation loss: 0.048849 Best loss: 0.031563 Accuracy: 98.91% 35 Validation loss: 0.049819 Best loss: 0.031563 Accuracy: 98.71% 36 Validation loss: 0.035035 Best loss: 0.031563 Accuracy: 99.26% 37 Validation loss: 0.054350 Best loss: 0.031563 Accuracy: 99.02% 38 Validation loss: 0.041283 Best loss: 0.031563 Accuracy: 99.10% 39 Validation loss: 0.034695 Best loss: 0.031563 Accuracy: 99.10% 40 Validation loss: 0.044970 Best loss: 0.031563 Accuracy: 99.14% 41 Validation loss: 0.029835 Best loss: 0.029835 Accuracy: 99.26% 42 Validation loss: 0.036198 Best loss: 0.029835 Accuracy: 99.14% 43 Validation loss: 0.039371 Best loss: 0.029835 Accuracy: 99.10% 44 Validation loss: 0.037529 Best loss: 0.029835 Accuracy: 99.26% 45 Validation loss: 0.040051 Best loss: 0.029835 Accuracy: 99.18% 46 Validation loss: 0.042828 Best loss: 0.029835 Accuracy: 99.18% 47 Validation loss: 0.044279 Best loss: 0.029835 Accuracy: 99.22% 48 Validation loss: 0.042722 Best loss: 0.029835 Accuracy: 99.26% 49 Validation loss: 0.040264 Best loss: 0.029835 Accuracy: 99.30% 50 Validation loss: 0.041836 Best loss: 0.029835 Accuracy: 99.22% 51 Validation loss: 0.042167 Best loss: 0.029835 Accuracy: 99.26% 52 Validation loss: 0.045520 Best loss: 0.029835 Accuracy: 99.14% 53 Validation loss: 0.047905 Best loss: 0.029835 Accuracy: 99.18% 54 Validation loss: 0.049705 Best loss: 0.029835 Accuracy: 99.18% 55 Validation loss: 0.050852 Best loss: 0.029835 Accuracy: 99.10% 56 Validation loss: 0.053921 Best loss: 0.029835 Accuracy: 99.06% 57 Validation loss: 0.056503 Best loss: 0.029835 Accuracy: 98.94% 58 Validation loss: 0.083642 Best loss: 0.029835 Accuracy: 98.67% 59 Validation loss: 0.068527 Best loss: 0.029835 Accuracy: 98.63% 60 Validation loss: 0.050438 Best loss: 0.029835 Accuracy: 98.91% 61 Validation loss: 0.051984 Best loss: 0.029835 Accuracy: 99.10% 62 Validation loss: 0.031038 Best loss: 0.029835 Accuracy: 99.34% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 22.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.066718 Best loss: 0.066718 Accuracy: 98.08% 1 Validation loss: 0.052706 Best loss: 0.052706 Accuracy: 98.32% 2 Validation loss: 0.034536 Best loss: 0.034536 Accuracy: 98.75% 3 Validation loss: 0.036595 Best loss: 0.034536 Accuracy: 98.83% 4 Validation loss: 0.049594 Best loss: 0.034536 Accuracy: 98.59% 5 Validation loss: 0.043853 Best loss: 0.034536 Accuracy: 98.67% 6 Validation loss: 0.047343 Best loss: 0.034536 Accuracy: 98.83% 7 Validation loss: 0.048611 Best loss: 0.034536 Accuracy: 98.83% 8 Validation loss: 0.048434 Best loss: 0.034536 Accuracy: 98.59% 9 Validation loss: 0.056742 Best loss: 0.034536 Accuracy: 98.67% 10 Validation loss: 0.044135 Best loss: 0.034536 Accuracy: 98.79% 11 Validation loss: 0.043949 Best loss: 0.034536 Accuracy: 98.98% 12 Validation loss: 0.051556 Best loss: 0.034536 Accuracy: 98.55% 13 Validation loss: 0.041360 Best loss: 0.034536 Accuracy: 99.06% 14 Validation loss: 0.042968 Best loss: 0.034536 Accuracy: 99.06% 15 Validation loss: 0.051872 Best loss: 0.034536 Accuracy: 98.79% 16 Validation loss: 0.047812 Best loss: 0.034536 Accuracy: 98.94% 17 Validation loss: 0.053140 Best loss: 0.034536 Accuracy: 98.75% 18 Validation loss: 0.052463 Best loss: 0.034536 Accuracy: 98.87% 19 Validation loss: 0.060392 Best loss: 0.034536 Accuracy: 98.51% 20 Validation loss: 0.046568 Best loss: 0.034536 Accuracy: 98.87% 21 Validation loss: 0.043291 Best loss: 0.034536 Accuracy: 98.98% 22 Validation loss: 0.048220 Best loss: 0.034536 Accuracy: 98.83% 23 Validation loss: 0.045609 Best loss: 0.034536 Accuracy: 98.79% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 9.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.064687 Best loss: 0.064687 Accuracy: 98.16% 1 Validation loss: 0.051339 Best loss: 0.051339 Accuracy: 98.51% 2 Validation loss: 0.042501 Best loss: 0.042501 Accuracy: 98.79% 3 Validation loss: 0.045774 Best loss: 0.042501 Accuracy: 98.40% 4 Validation loss: 0.049369 Best loss: 0.042501 Accuracy: 98.71% 5 Validation loss: 0.064206 Best loss: 0.042501 Accuracy: 98.28% 6 Validation loss: 0.065411 Best loss: 0.042501 Accuracy: 98.28% 7 Validation loss: 0.065870 Best loss: 0.042501 Accuracy: 98.24% 8 Validation loss: 0.045822 Best loss: 0.042501 Accuracy: 98.75% 9 Validation loss: 0.048240 Best loss: 0.042501 Accuracy: 98.98% 10 Validation loss: 0.046939 Best loss: 0.042501 Accuracy: 98.83% 11 Validation loss: 0.056476 Best loss: 0.042501 Accuracy: 98.83% 12 Validation loss: 0.042997 Best loss: 0.042501 Accuracy: 99.10% 13 Validation loss: 0.052289 Best loss: 0.042501 Accuracy: 98.91% 14 Validation loss: 0.036608 Best loss: 0.036608 Accuracy: 99.18% 15 Validation loss: 0.037099 Best loss: 0.036608 Accuracy: 99.06% 16 Validation loss: 0.060304 Best loss: 0.036608 Accuracy: 98.71% 17 Validation loss: 0.043546 Best loss: 0.036608 Accuracy: 99.06% 18 Validation loss: 0.054232 Best loss: 0.036608 Accuracy: 98.98% 19 Validation loss: 0.058274 Best loss: 0.036608 Accuracy: 98.83% 20 Validation loss: 0.062562 Best loss: 0.036608 Accuracy: 98.79% 21 Validation loss: 0.056829 Best loss: 0.036608 Accuracy: 98.79% 22 Validation loss: 0.052649 Best loss: 0.036608 Accuracy: 98.87% 23 Validation loss: 0.049380 Best loss: 0.036608 Accuracy: 98.87% 24 Validation loss: 0.050732 Best loss: 0.036608 Accuracy: 98.83% 25 Validation loss: 0.039589 Best loss: 0.036608 Accuracy: 99.18% 26 Validation loss: 0.039263 Best loss: 0.036608 Accuracy: 99.02% 27 Validation loss: 0.039192 Best loss: 0.036608 Accuracy: 98.87% 28 Validation loss: 0.038463 Best loss: 0.036608 Accuracy: 99.22% 29 Validation loss: 0.037691 Best loss: 0.036608 Accuracy: 99.10% 30 Validation loss: 0.039472 Best loss: 0.036608 Accuracy: 99.22% 31 Validation loss: 0.038419 Best loss: 0.036608 Accuracy: 99.06% 32 Validation loss: 0.039220 Best loss: 0.036608 Accuracy: 99.22% 33 Validation loss: 0.038718 Best loss: 0.036608 Accuracy: 99.26% 34 Validation loss: 0.035110 Best loss: 0.035110 Accuracy: 99.14% 35 Validation loss: 0.053937 Best loss: 0.035110 Accuracy: 98.79% 36 Validation loss: 0.048376 Best loss: 0.035110 Accuracy: 98.94% 37 Validation loss: 0.069572 Best loss: 0.035110 Accuracy: 98.63% 38 Validation loss: 0.054074 Best loss: 0.035110 Accuracy: 98.94% 39 Validation loss: 0.052425 Best loss: 0.035110 Accuracy: 98.87% 40 Validation loss: 0.045873 Best loss: 0.035110 Accuracy: 98.98% 41 Validation loss: 0.039603 Best loss: 0.035110 Accuracy: 99.02% 42 Validation loss: 0.038031 Best loss: 0.035110 Accuracy: 99.02% 43 Validation loss: 0.040737 Best loss: 0.035110 Accuracy: 99.18% 44 Validation loss: 0.064919 Best loss: 0.035110 Accuracy: 98.71% 45 Validation loss: 0.042356 Best loss: 0.035110 Accuracy: 99.06% 46 Validation loss: 0.041273 Best loss: 0.035110 Accuracy: 99.18% 47 Validation loss: 0.049639 Best loss: 0.035110 Accuracy: 98.87% 48 Validation loss: 0.041741 Best loss: 0.035110 Accuracy: 99.14% 49 Validation loss: 0.034264 Best loss: 0.034264 Accuracy: 99.26% 50 Validation loss: 0.068849 Best loss: 0.034264 Accuracy: 98.98% 51 Validation loss: 0.066571 Best loss: 0.034264 Accuracy: 98.59% 52 Validation loss: 0.049398 Best loss: 0.034264 Accuracy: 98.83% 53 Validation loss: 0.033788 Best loss: 0.033788 Accuracy: 99.18% 54 Validation loss: 0.040954 Best loss: 0.033788 Accuracy: 99.10% 55 Validation loss: 0.054927 Best loss: 0.033788 Accuracy: 98.79% 56 Validation loss: 0.040091 Best loss: 0.033788 Accuracy: 99.18% 57 Validation loss: 0.060415 Best loss: 0.033788 Accuracy: 98.75% 58 Validation loss: 0.047220 Best loss: 0.033788 Accuracy: 98.98% 59 Validation loss: 0.055318 Best loss: 0.033788 Accuracy: 99.06% 60 Validation loss: 0.048520 Best loss: 0.033788 Accuracy: 99.22% 61 Validation loss: 0.047132 Best loss: 0.033788 Accuracy: 99.30% 62 Validation loss: 0.046482 Best loss: 0.033788 Accuracy: 99.30% 63 Validation loss: 0.048665 Best loss: 0.033788 Accuracy: 99.30% 64 Validation loss: 0.046323 Best loss: 0.033788 Accuracy: 99.26% 65 Validation loss: 0.060230 Best loss: 0.033788 Accuracy: 98.79% 66 Validation loss: 0.042665 Best loss: 0.033788 Accuracy: 99.06% 67 Validation loss: 0.046175 Best loss: 0.033788 Accuracy: 99.14% 68 Validation loss: 0.049718 Best loss: 0.033788 Accuracy: 98.79% 69 Validation loss: 0.074584 Best loss: 0.033788 Accuracy: 98.51% 70 Validation loss: 0.058340 Best loss: 0.033788 Accuracy: 98.87% 71 Validation loss: 0.054455 Best loss: 0.033788 Accuracy: 98.79% 72 Validation loss: 0.068880 Best loss: 0.033788 Accuracy: 98.63% 73 Validation loss: 0.058834 Best loss: 0.033788 Accuracy: 98.87% 74 Validation loss: 0.051500 Best loss: 0.033788 Accuracy: 98.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 26.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.117541 Best loss: 0.117541 Accuracy: 96.48% 1 Validation loss: 0.074757 Best loss: 0.074757 Accuracy: 97.85% 2 Validation loss: 0.066205 Best loss: 0.066205 Accuracy: 98.08% 3 Validation loss: 0.079499 Best loss: 0.066205 Accuracy: 97.97% 4 Validation loss: 0.058983 Best loss: 0.058983 Accuracy: 98.40% 5 Validation loss: 0.062375 Best loss: 0.058983 Accuracy: 98.63% 6 Validation loss: 0.089536 Best loss: 0.058983 Accuracy: 98.28% 7 Validation loss: 0.077251 Best loss: 0.058983 Accuracy: 98.55% 8 Validation loss: 0.049218 Best loss: 0.049218 Accuracy: 98.75% 9 Validation loss: 0.064269 Best loss: 0.049218 Accuracy: 98.59% 10 Validation loss: 0.069041 Best loss: 0.049218 Accuracy: 98.44% 11 Validation loss: 0.050668 Best loss: 0.049218 Accuracy: 98.63% 12 Validation loss: 0.051518 Best loss: 0.049218 Accuracy: 98.59% 13 Validation loss: 1.329084 Best loss: 0.049218 Accuracy: 84.44% 14 Validation loss: 0.092132 Best loss: 0.049218 Accuracy: 98.01% 15 Validation loss: 0.055490 Best loss: 0.049218 Accuracy: 98.71% 16 Validation loss: 0.065910 Best loss: 0.049218 Accuracy: 98.59% 17 Validation loss: 0.063333 Best loss: 0.049218 Accuracy: 98.55% 18 Validation loss: 0.296028 Best loss: 0.049218 Accuracy: 96.36% 19 Validation loss: 0.067347 Best loss: 0.049218 Accuracy: 98.51% 20 Validation loss: 0.062401 Best loss: 0.049218 Accuracy: 98.94% 21 Validation loss: 0.095642 Best loss: 0.049218 Accuracy: 98.71% 22 Validation loss: 0.104568 Best loss: 0.049218 Accuracy: 98.32% 23 Validation loss: 0.075313 Best loss: 0.049218 Accuracy: 98.83% 24 Validation loss: 0.077622 Best loss: 0.049218 Accuracy: 98.71% 25 Validation loss: 0.062709 Best loss: 0.049218 Accuracy: 98.98% 26 Validation loss: 0.085009 Best loss: 0.049218 Accuracy: 98.51% 27 Validation loss: 0.158695 Best loss: 0.049218 Accuracy: 98.40% 28 Validation loss: 0.292002 Best loss: 0.049218 Accuracy: 96.33% 29 Validation loss: 0.069179 Best loss: 0.049218 Accuracy: 98.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.104882 Best loss: 0.104882 Accuracy: 96.91% 1 Validation loss: 0.106673 Best loss: 0.104882 Accuracy: 96.99% 2 Validation loss: 0.084040 Best loss: 0.084040 Accuracy: 97.89% 3 Validation loss: 0.063556 Best loss: 0.063556 Accuracy: 98.08% 4 Validation loss: 0.061092 Best loss: 0.061092 Accuracy: 98.40% 5 Validation loss: 0.096903 Best loss: 0.061092 Accuracy: 97.22% 6 Validation loss: 0.061854 Best loss: 0.061092 Accuracy: 98.48% 7 Validation loss: 0.059892 Best loss: 0.059892 Accuracy: 98.28% 8 Validation loss: 0.060223 Best loss: 0.059892 Accuracy: 98.32% 9 Validation loss: 0.071010 Best loss: 0.059892 Accuracy: 98.36% 10 Validation loss: 0.070598 Best loss: 0.059892 Accuracy: 98.24% 11 Validation loss: 0.094459 Best loss: 0.059892 Accuracy: 97.77% 12 Validation loss: 0.112733 Best loss: 0.059892 Accuracy: 97.38% 13 Validation loss: 0.060980 Best loss: 0.059892 Accuracy: 98.67% 14 Validation loss: 0.065557 Best loss: 0.059892 Accuracy: 98.51% 15 Validation loss: 0.092551 Best loss: 0.059892 Accuracy: 98.20% 16 Validation loss: 0.095083 Best loss: 0.059892 Accuracy: 97.85% 17 Validation loss: 0.060173 Best loss: 0.059892 Accuracy: 98.59% 18 Validation loss: 0.093357 Best loss: 0.059892 Accuracy: 98.28% 19 Validation loss: 0.073536 Best loss: 0.059892 Accuracy: 98.59% 20 Validation loss: 0.059048 Best loss: 0.059048 Accuracy: 98.63% 21 Validation loss: 0.082017 Best loss: 0.059048 Accuracy: 98.32% 22 Validation loss: 0.086512 Best loss: 0.059048 Accuracy: 98.67% 23 Validation loss: 0.052362 Best loss: 0.052362 Accuracy: 98.94% 24 Validation loss: 0.107535 Best loss: 0.052362 Accuracy: 98.48% 25 Validation loss: 0.102378 Best loss: 0.052362 Accuracy: 98.12% 26 Validation loss: 0.161408 Best loss: 0.052362 Accuracy: 98.44% 27 Validation loss: 0.081292 Best loss: 0.052362 Accuracy: 98.83% 28 Validation loss: 0.062219 Best loss: 0.052362 Accuracy: 98.79% 29 Validation loss: 0.068106 Best loss: 0.052362 Accuracy: 98.91% 30 Validation loss: 0.090153 Best loss: 0.052362 Accuracy: 98.67% 31 Validation loss: 0.053843 Best loss: 0.052362 Accuracy: 98.91% 32 Validation loss: 0.059875 Best loss: 0.052362 Accuracy: 98.67% 33 Validation loss: 0.124206 Best loss: 0.052362 Accuracy: 98.67% 34 Validation loss: 0.081096 Best loss: 0.052362 Accuracy: 98.87% 35 Validation loss: 0.122021 Best loss: 0.052362 Accuracy: 98.32% 36 Validation loss: 0.150488 Best loss: 0.052362 Accuracy: 98.16% 37 Validation loss: 0.072864 Best loss: 0.052362 Accuracy: 98.87% 38 Validation loss: 0.108794 Best loss: 0.052362 Accuracy: 98.55% 39 Validation loss: 0.213613 Best loss: 0.052362 Accuracy: 98.28% 40 Validation loss: 0.083839 Best loss: 0.052362 Accuracy: 98.63% 41 Validation loss: 0.104061 Best loss: 0.052362 Accuracy: 98.71% 42 Validation loss: 0.076792 Best loss: 0.052362 Accuracy: 98.94% 43 Validation loss: 0.104143 Best loss: 0.052362 Accuracy: 98.75% 44 Validation loss: 0.111759 Best loss: 0.052362 Accuracy: 98.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.9min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.171695 Best loss: 0.171695 Accuracy: 95.70% 1 Validation loss: 0.095867 Best loss: 0.095867 Accuracy: 97.11% 2 Validation loss: 0.062387 Best loss: 0.062387 Accuracy: 98.08% 3 Validation loss: 0.077939 Best loss: 0.062387 Accuracy: 98.12% 4 Validation loss: 0.080375 Best loss: 0.062387 Accuracy: 98.01% 5 Validation loss: 0.067616 Best loss: 0.062387 Accuracy: 97.93% 6 Validation loss: 0.211783 Best loss: 0.062387 Accuracy: 95.93% 7 Validation loss: 0.055902 Best loss: 0.055902 Accuracy: 98.51% 8 Validation loss: 0.036893 Best loss: 0.036893 Accuracy: 98.98% 9 Validation loss: 0.089392 Best loss: 0.036893 Accuracy: 97.77% 10 Validation loss: 0.043042 Best loss: 0.036893 Accuracy: 98.91% 11 Validation loss: 0.049621 Best loss: 0.036893 Accuracy: 98.71% 12 Validation loss: 0.042615 Best loss: 0.036893 Accuracy: 98.44% 13 Validation loss: 0.040890 Best loss: 0.036893 Accuracy: 98.83% 14 Validation loss: 0.264979 Best loss: 0.036893 Accuracy: 94.57% 15 Validation loss: 0.085482 Best loss: 0.036893 Accuracy: 98.75% 16 Validation loss: 0.075842 Best loss: 0.036893 Accuracy: 98.48% 17 Validation loss: 0.051864 Best loss: 0.036893 Accuracy: 98.87% 18 Validation loss: 0.075675 Best loss: 0.036893 Accuracy: 98.44% 19 Validation loss: 0.046044 Best loss: 0.036893 Accuracy: 99.06% 20 Validation loss: 0.082570 Best loss: 0.036893 Accuracy: 98.63% 21 Validation loss: 0.076438 Best loss: 0.036893 Accuracy: 98.40% 22 Validation loss: 0.057702 Best loss: 0.036893 Accuracy: 98.98% 23 Validation loss: 0.042962 Best loss: 0.036893 Accuracy: 99.34% 24 Validation loss: 0.066657 Best loss: 0.036893 Accuracy: 98.79% 25 Validation loss: 0.078532 Best loss: 0.036893 Accuracy: 98.67% 26 Validation loss: 0.042448 Best loss: 0.036893 Accuracy: 99.02% 27 Validation loss: 0.061020 Best loss: 0.036893 Accuracy: 98.91% 28 Validation loss: 0.039687 Best loss: 0.036893 Accuracy: 99.14% 29 Validation loss: 0.055059 Best loss: 0.036893 Accuracy: 98.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.153621 Best loss: 0.153621 Accuracy: 97.07% 1 Validation loss: 0.097455 Best loss: 0.097455 Accuracy: 97.30% 2 Validation loss: 0.062807 Best loss: 0.062807 Accuracy: 98.28% 3 Validation loss: 0.060732 Best loss: 0.060732 Accuracy: 98.16% 4 Validation loss: 0.053661 Best loss: 0.053661 Accuracy: 98.12% 5 Validation loss: 0.064446 Best loss: 0.053661 Accuracy: 98.48% 6 Validation loss: 0.091685 Best loss: 0.053661 Accuracy: 98.01% 7 Validation loss: 0.076015 Best loss: 0.053661 Accuracy: 98.24% 8 Validation loss: 0.058148 Best loss: 0.053661 Accuracy: 98.20% 9 Validation loss: 0.067005 Best loss: 0.053661 Accuracy: 98.55% 10 Validation loss: 0.056755 Best loss: 0.053661 Accuracy: 99.02% 11 Validation loss: 0.079325 Best loss: 0.053661 Accuracy: 98.40% 12 Validation loss: 0.078098 Best loss: 0.053661 Accuracy: 98.40% 13 Validation loss: 0.079808 Best loss: 0.053661 Accuracy: 98.44% 14 Validation loss: 0.061565 Best loss: 0.053661 Accuracy: 98.51% 15 Validation loss: 0.079895 Best loss: 0.053661 Accuracy: 98.48% 16 Validation loss: 0.062152 Best loss: 0.053661 Accuracy: 98.48% 17 Validation loss: 0.076255 Best loss: 0.053661 Accuracy: 98.28% 18 Validation loss: 0.075098 Best loss: 0.053661 Accuracy: 98.59% 19 Validation loss: 0.058472 Best loss: 0.053661 Accuracy: 98.67% 20 Validation loss: 0.062936 Best loss: 0.053661 Accuracy: 98.63% 21 Validation loss: 0.050526 Best loss: 0.050526 Accuracy: 98.87% 22 Validation loss: 0.055341 Best loss: 0.050526 Accuracy: 98.94% 23 Validation loss: 0.060796 Best loss: 0.050526 Accuracy: 98.91% 24 Validation loss: 0.049021 Best loss: 0.049021 Accuracy: 98.98% 25 Validation loss: 0.074429 Best loss: 0.049021 Accuracy: 98.71% 26 Validation loss: 0.069348 Best loss: 0.049021 Accuracy: 98.59% 27 Validation loss: 0.067372 Best loss: 0.049021 Accuracy: 98.36% 28 Validation loss: 0.059254 Best loss: 0.049021 Accuracy: 98.79% 29 Validation loss: 0.055321 Best loss: 0.049021 Accuracy: 98.79% 30 Validation loss: 0.061204 Best loss: 0.049021 Accuracy: 98.71% 31 Validation loss: 0.058381 Best loss: 0.049021 Accuracy: 98.75% 32 Validation loss: 0.051402 Best loss: 0.049021 Accuracy: 98.87% 33 Validation loss: 0.055209 Best loss: 0.049021 Accuracy: 99.02% 34 Validation loss: 0.051796 Best loss: 0.049021 Accuracy: 99.06% 35 Validation loss: 0.050974 Best loss: 0.049021 Accuracy: 98.94% 36 Validation loss: 0.048991 Best loss: 0.048991 Accuracy: 99.02% 37 Validation loss: 0.048650 Best loss: 0.048650 Accuracy: 99.02% 38 Validation loss: 0.047571 Best loss: 0.047571 Accuracy: 99.14% 39 Validation loss: 0.045871 Best loss: 0.045871 Accuracy: 99.06% 40 Validation loss: 0.072951 Best loss: 0.045871 Accuracy: 98.75% 41 Validation loss: 0.057602 Best loss: 0.045871 Accuracy: 99.10% 42 Validation loss: 0.115990 Best loss: 0.045871 Accuracy: 98.05% 43 Validation loss: 0.075653 Best loss: 0.045871 Accuracy: 98.55% 44 Validation loss: 0.069671 Best loss: 0.045871 Accuracy: 98.32% 45 Validation loss: 0.074374 Best loss: 0.045871 Accuracy: 98.36% 46 Validation loss: 0.060073 Best loss: 0.045871 Accuracy: 98.91% 47 Validation loss: 0.068678 Best loss: 0.045871 Accuracy: 98.67% 48 Validation loss: 0.079527 Best loss: 0.045871 Accuracy: 98.63% 49 Validation loss: 0.096985 Best loss: 0.045871 Accuracy: 98.08% 50 Validation loss: 0.056538 Best loss: 0.045871 Accuracy: 98.91% 51 Validation loss: 0.057080 Best loss: 0.045871 Accuracy: 98.91% 52 Validation loss: 0.057977 Best loss: 0.045871 Accuracy: 98.79% 53 Validation loss: 0.059260 Best loss: 0.045871 Accuracy: 98.83% 54 Validation loss: 0.060183 Best loss: 0.045871 Accuracy: 98.91% 55 Validation loss: 0.070792 Best loss: 0.045871 Accuracy: 98.71% 56 Validation loss: 0.053380 Best loss: 0.045871 Accuracy: 99.02% 57 Validation loss: 0.048671 Best loss: 0.045871 Accuracy: 99.14% 58 Validation loss: 0.050407 Best loss: 0.045871 Accuracy: 99.06% 59 Validation loss: 0.048734 Best loss: 0.045871 Accuracy: 99.10% 60 Validation loss: 0.045989 Best loss: 0.045871 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 20.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.142051 Best loss: 0.142051 Accuracy: 97.03% 1 Validation loss: 0.082228 Best loss: 0.082228 Accuracy: 98.20% 2 Validation loss: 0.050449 Best loss: 0.050449 Accuracy: 98.48% 3 Validation loss: 0.060512 Best loss: 0.050449 Accuracy: 98.40% 4 Validation loss: 0.064697 Best loss: 0.050449 Accuracy: 98.55% 5 Validation loss: 0.057914 Best loss: 0.050449 Accuracy: 98.40% 6 Validation loss: 0.053191 Best loss: 0.050449 Accuracy: 98.71% 7 Validation loss: 0.062977 Best loss: 0.050449 Accuracy: 98.59% 8 Validation loss: 0.051546 Best loss: 0.050449 Accuracy: 98.75% 9 Validation loss: 0.065491 Best loss: 0.050449 Accuracy: 98.59% 10 Validation loss: 0.047255 Best loss: 0.047255 Accuracy: 98.71% 11 Validation loss: 0.051346 Best loss: 0.047255 Accuracy: 98.71% 12 Validation loss: 0.059480 Best loss: 0.047255 Accuracy: 98.67% 13 Validation loss: 0.055278 Best loss: 0.047255 Accuracy: 98.59% 14 Validation loss: 0.057548 Best loss: 0.047255 Accuracy: 98.59% 15 Validation loss: 0.054354 Best loss: 0.047255 Accuracy: 98.63% 16 Validation loss: 0.077575 Best loss: 0.047255 Accuracy: 98.08% 17 Validation loss: 0.055893 Best loss: 0.047255 Accuracy: 98.79% 18 Validation loss: 0.055464 Best loss: 0.047255 Accuracy: 98.75% 19 Validation loss: 0.060377 Best loss: 0.047255 Accuracy: 98.71% 20 Validation loss: 0.044530 Best loss: 0.044530 Accuracy: 99.06% 21 Validation loss: 0.053282 Best loss: 0.044530 Accuracy: 98.91% 22 Validation loss: 0.067141 Best loss: 0.044530 Accuracy: 98.48% 23 Validation loss: 0.048836 Best loss: 0.044530 Accuracy: 98.83% 24 Validation loss: 0.038790 Best loss: 0.038790 Accuracy: 99.14% 25 Validation loss: 0.066960 Best loss: 0.038790 Accuracy: 98.55% 26 Validation loss: 0.049634 Best loss: 0.038790 Accuracy: 98.79% 27 Validation loss: 0.057723 Best loss: 0.038790 Accuracy: 98.75% 28 Validation loss: 0.058969 Best loss: 0.038790 Accuracy: 98.79% 29 Validation loss: 0.048606 Best loss: 0.038790 Accuracy: 98.67% 30 Validation loss: 0.087307 Best loss: 0.038790 Accuracy: 98.71% 31 Validation loss: 0.067172 Best loss: 0.038790 Accuracy: 98.48% 32 Validation loss: 0.074619 Best loss: 0.038790 Accuracy: 98.55% 33 Validation loss: 0.074017 Best loss: 0.038790 Accuracy: 98.55% 34 Validation loss: 0.069904 Best loss: 0.038790 Accuracy: 98.55% 35 Validation loss: 0.085311 Best loss: 0.038790 Accuracy: 98.40% 36 Validation loss: 0.050216 Best loss: 0.038790 Accuracy: 99.06% 37 Validation loss: 0.057665 Best loss: 0.038790 Accuracy: 98.83% 38 Validation loss: 0.048263 Best loss: 0.038790 Accuracy: 98.98% 39 Validation loss: 0.048676 Best loss: 0.038790 Accuracy: 99.02% 40 Validation loss: 0.054454 Best loss: 0.038790 Accuracy: 98.94% 41 Validation loss: 0.061809 Best loss: 0.038790 Accuracy: 98.83% 42 Validation loss: 0.060230 Best loss: 0.038790 Accuracy: 98.75% 43 Validation loss: 0.055911 Best loss: 0.038790 Accuracy: 98.94% 44 Validation loss: 0.066906 Best loss: 0.038790 Accuracy: 98.59% 45 Validation loss: 0.062493 Best loss: 0.038790 Accuracy: 98.51% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 17.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02 0 Validation loss: 0.144453 Best loss: 0.144453 Accuracy: 97.22% 1 Validation loss: 0.073664 Best loss: 0.073664 Accuracy: 98.16% 2 Validation loss: 0.064706 Best loss: 0.064706 Accuracy: 98.44% 3 Validation loss: 0.057106 Best loss: 0.057106 Accuracy: 98.63% 4 Validation loss: 0.066407 Best loss: 0.057106 Accuracy: 98.08% 5 Validation loss: 0.052937 Best loss: 0.052937 Accuracy: 98.67% 6 Validation loss: 0.058683 Best loss: 0.052937 Accuracy: 98.79% 7 Validation loss: 0.082794 Best loss: 0.052937 Accuracy: 98.28% 8 Validation loss: 0.080461 Best loss: 0.052937 Accuracy: 98.16% 9 Validation loss: 0.053259 Best loss: 0.052937 Accuracy: 98.55% 10 Validation loss: 0.059246 Best loss: 0.052937 Accuracy: 98.55% 11 Validation loss: 0.057574 Best loss: 0.052937 Accuracy: 98.44% 12 Validation loss: 0.049159 Best loss: 0.049159 Accuracy: 98.91% 13 Validation loss: 0.052442 Best loss: 0.049159 Accuracy: 98.87% 14 Validation loss: 0.062400 Best loss: 0.049159 Accuracy: 98.59% 15 Validation loss: 0.056643 Best loss: 0.049159 Accuracy: 98.59% 16 Validation loss: 0.056916 Best loss: 0.049159 Accuracy: 98.67% 17 Validation loss: 0.059061 Best loss: 0.049159 Accuracy: 98.91% 18 Validation loss: 0.066379 Best loss: 0.049159 Accuracy: 98.63% 19 Validation loss: 0.052594 Best loss: 0.049159 Accuracy: 98.87% 20 Validation loss: 0.062033 Best loss: 0.049159 Accuracy: 98.63% 21 Validation loss: 0.065245 Best loss: 0.049159 Accuracy: 98.71% 22 Validation loss: 0.057142 Best loss: 0.049159 Accuracy: 98.94% 23 Validation loss: 0.055719 Best loss: 0.049159 Accuracy: 98.94% 24 Validation loss: 0.098378 Best loss: 0.049159 Accuracy: 98.12% 25 Validation loss: 0.065951 Best loss: 0.049159 Accuracy: 98.63% 26 Validation loss: 0.064796 Best loss: 0.049159 Accuracy: 98.79% 27 Validation loss: 0.057005 Best loss: 0.049159 Accuracy: 98.87% 28 Validation loss: 0.078843 Best loss: 0.049159 Accuracy: 98.36% 29 Validation loss: 0.052311 Best loss: 0.049159 Accuracy: 98.98% 30 Validation loss: 0.065484 Best loss: 0.049159 Accuracy: 98.63% 31 Validation loss: 0.063528 Best loss: 0.049159 Accuracy: 98.83% 32 Validation loss: 0.049519 Best loss: 0.049159 Accuracy: 98.94% 33 Validation loss: 0.048161 Best loss: 0.048161 Accuracy: 99.10% 34 Validation loss: 0.049902 Best loss: 0.048161 Accuracy: 98.87% 35 Validation loss: 0.046750 Best loss: 0.046750 Accuracy: 99.18% 36 Validation loss: 0.094648 Best loss: 0.046750 Accuracy: 98.48% 37 Validation loss: 0.075936 Best loss: 0.046750 Accuracy: 98.44% 38 Validation loss: 0.064824 Best loss: 0.046750 Accuracy: 98.55% 39 Validation loss: 0.049341 Best loss: 0.046750 Accuracy: 98.83% 40 Validation loss: 0.061405 Best loss: 0.046750 Accuracy: 98.83% 41 Validation loss: 0.055854 Best loss: 0.046750 Accuracy: 98.91% 42 Validation loss: 0.060956 Best loss: 0.046750 Accuracy: 98.83% 43 Validation loss: 0.048781 Best loss: 0.046750 Accuracy: 98.91% 44 Validation loss: 0.078308 Best loss: 0.046750 Accuracy: 98.67% 45 Validation loss: 0.054695 Best loss: 0.046750 Accuracy: 98.83% 46 Validation loss: 0.043962 Best loss: 0.043962 Accuracy: 99.02% 47 Validation loss: 0.057880 Best loss: 0.043962 Accuracy: 98.91% 48 Validation loss: 0.061129 Best loss: 0.043962 Accuracy: 98.83% 49 Validation loss: 0.058749 Best loss: 0.043962 Accuracy: 98.59% 50 Validation loss: 0.068124 Best loss: 0.043962 Accuracy: 98.83% 51 Validation loss: 0.062610 Best loss: 0.043962 Accuracy: 98.75% 52 Validation loss: 0.064342 Best loss: 0.043962 Accuracy: 98.83% 53 Validation loss: 0.067911 Best loss: 0.043962 Accuracy: 98.87% 54 Validation loss: 0.066347 Best loss: 0.043962 Accuracy: 98.75% 55 Validation loss: 0.066692 Best loss: 0.043962 Accuracy: 98.71% 56 Validation loss: 0.050424 Best loss: 0.043962 Accuracy: 98.87% 57 Validation loss: 0.045474 Best loss: 0.043962 Accuracy: 98.55% 58 Validation loss: 0.046317 Best loss: 0.043962 Accuracy: 98.91% 59 Validation loss: 0.052503 Best loss: 0.043962 Accuracy: 98.87% 60 Validation loss: 0.041963 Best loss: 0.041963 Accuracy: 99.02% 61 Validation loss: 0.049972 Best loss: 0.041963 Accuracy: 99.06% 62 Validation loss: 0.057595 Best loss: 0.041963 Accuracy: 98.94% 63 Validation loss: 0.066497 Best loss: 0.041963 Accuracy: 98.75% 64 Validation loss: 0.064887 Best loss: 0.041963 Accuracy: 98.79% 65 Validation loss: 0.063013 Best loss: 0.041963 Accuracy: 98.98% 66 Validation loss: 0.070365 Best loss: 0.041963 Accuracy: 98.67% 67 Validation loss: 0.064260 Best loss: 0.041963 Accuracy: 98.87% 68 Validation loss: 0.051648 Best loss: 0.041963 Accuracy: 98.87% 69 Validation loss: 0.051070 Best loss: 0.041963 Accuracy: 98.91% 70 Validation loss: 0.051334 Best loss: 0.041963 Accuracy: 98.87% 71 Validation loss: 0.053668 Best loss: 0.041963 Accuracy: 98.83% 72 Validation loss: 0.086659 Best loss: 0.041963 Accuracy: 98.20% 73 Validation loss: 0.078000 Best loss: 0.041963 Accuracy: 98.71% 74 Validation loss: 0.064688 Best loss: 0.041963 Accuracy: 98.94% 75 Validation loss: 0.077326 Best loss: 0.041963 Accuracy: 98.63% 76 Validation loss: 0.065047 Best loss: 0.041963 Accuracy: 98.79% 77 Validation loss: 0.056128 Best loss: 0.041963 Accuracy: 99.06% 78 Validation loss: 0.062973 Best loss: 0.041963 Accuracy: 98.87% 79 Validation loss: 0.059021 Best loss: 0.041963 Accuracy: 98.94% 80 Validation loss: 0.103144 Best loss: 0.041963 Accuracy: 97.93% 81 Validation loss: 0.073427 Best loss: 0.041963 Accuracy: 98.67% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=50, batch_size=500, batch_norm_momentum=0.98, learning_rate=0.02, total= 27.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.103251 Best loss: 0.103251 Accuracy: 98.05% 1 Validation loss: 0.098870 Best loss: 0.098870 Accuracy: 97.69% 2 Validation loss: 0.077983 Best loss: 0.077983 Accuracy: 98.32% 3 Validation loss: 0.069740 Best loss: 0.069740 Accuracy: 98.16% 4 Validation loss: 0.072758 Best loss: 0.069740 Accuracy: 98.32% 5 Validation loss: 0.054964 Best loss: 0.054964 Accuracy: 98.59% 6 Validation loss: 0.052166 Best loss: 0.052166 Accuracy: 98.75% 7 Validation loss: 0.043800 Best loss: 0.043800 Accuracy: 98.91% 8 Validation loss: 0.036479 Best loss: 0.036479 Accuracy: 98.94% 9 Validation loss: 0.070414 Best loss: 0.036479 Accuracy: 98.63% 10 Validation loss: 0.050572 Best loss: 0.036479 Accuracy: 98.67% 11 Validation loss: 0.038249 Best loss: 0.036479 Accuracy: 99.26% 12 Validation loss: 0.060499 Best loss: 0.036479 Accuracy: 98.71% 13 Validation loss: 0.059860 Best loss: 0.036479 Accuracy: 98.75% 14 Validation loss: 0.050868 Best loss: 0.036479 Accuracy: 98.67% 15 Validation loss: 0.043053 Best loss: 0.036479 Accuracy: 99.10% 16 Validation loss: 0.055316 Best loss: 0.036479 Accuracy: 99.10% 17 Validation loss: 0.045394 Best loss: 0.036479 Accuracy: 99.14% 18 Validation loss: 0.048286 Best loss: 0.036479 Accuracy: 99.06% 19 Validation loss: 0.060105 Best loss: 0.036479 Accuracy: 98.91% 20 Validation loss: 0.048416 Best loss: 0.036479 Accuracy: 99.02% 21 Validation loss: 0.074017 Best loss: 0.036479 Accuracy: 98.83% 22 Validation loss: 0.043033 Best loss: 0.036479 Accuracy: 98.98% 23 Validation loss: 0.049829 Best loss: 0.036479 Accuracy: 98.98% 24 Validation loss: 0.043462 Best loss: 0.036479 Accuracy: 99.14% 25 Validation loss: 0.057393 Best loss: 0.036479 Accuracy: 98.83% 26 Validation loss: 0.032122 Best loss: 0.032122 Accuracy: 99.30% 27 Validation loss: 0.044333 Best loss: 0.032122 Accuracy: 99.22% 28 Validation loss: 0.070130 Best loss: 0.032122 Accuracy: 98.67% 29 Validation loss: 0.053781 Best loss: 0.032122 Accuracy: 98.94% 30 Validation loss: 0.056610 Best loss: 0.032122 Accuracy: 99.02% 31 Validation loss: 0.040008 Best loss: 0.032122 Accuracy: 99.22% 32 Validation loss: 0.048776 Best loss: 0.032122 Accuracy: 99.10% 33 Validation loss: 0.058192 Best loss: 0.032122 Accuracy: 98.91% 34 Validation loss: 0.038817 Best loss: 0.032122 Accuracy: 99.22% 35 Validation loss: 0.050526 Best loss: 0.032122 Accuracy: 99.10% 36 Validation loss: 0.032229 Best loss: 0.032122 Accuracy: 99.14% 37 Validation loss: 0.121715 Best loss: 0.032122 Accuracy: 98.24% 38 Validation loss: 0.034044 Best loss: 0.032122 Accuracy: 99.30% 39 Validation loss: 0.048214 Best loss: 0.032122 Accuracy: 99.22% 40 Validation loss: 0.050738 Best loss: 0.032122 Accuracy: 98.87% 41 Validation loss: 0.046253 Best loss: 0.032122 Accuracy: 99.06% 42 Validation loss: 0.039627 Best loss: 0.032122 Accuracy: 99.14% 43 Validation loss: 0.050353 Best loss: 0.032122 Accuracy: 99.02% 44 Validation loss: 0.045985 Best loss: 0.032122 Accuracy: 99.10% 45 Validation loss: 0.035592 Best loss: 0.032122 Accuracy: 99.10% 46 Validation loss: 0.039247 Best loss: 0.032122 Accuracy: 99.26% 47 Validation loss: 0.037231 Best loss: 0.032122 Accuracy: 99.26% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.122132 Best loss: 0.122132 Accuracy: 97.38% 1 Validation loss: 0.077134 Best loss: 0.077134 Accuracy: 98.16% 2 Validation loss: 0.047492 Best loss: 0.047492 Accuracy: 98.71% 3 Validation loss: 0.063185 Best loss: 0.047492 Accuracy: 98.48% 4 Validation loss: 0.089423 Best loss: 0.047492 Accuracy: 98.24% 5 Validation loss: 0.040571 Best loss: 0.040571 Accuracy: 98.98% 6 Validation loss: 0.052551 Best loss: 0.040571 Accuracy: 98.71% 7 Validation loss: 0.033894 Best loss: 0.033894 Accuracy: 99.18% 8 Validation loss: 0.063203 Best loss: 0.033894 Accuracy: 98.63% 9 Validation loss: 0.048098 Best loss: 0.033894 Accuracy: 98.94% 10 Validation loss: 0.048439 Best loss: 0.033894 Accuracy: 99.02% 11 Validation loss: 0.050621 Best loss: 0.033894 Accuracy: 98.83% 12 Validation loss: 0.051986 Best loss: 0.033894 Accuracy: 98.91% 13 Validation loss: 0.036202 Best loss: 0.033894 Accuracy: 98.91% 14 Validation loss: 0.039310 Best loss: 0.033894 Accuracy: 99.06% 15 Validation loss: 0.056590 Best loss: 0.033894 Accuracy: 98.75% 16 Validation loss: 0.036199 Best loss: 0.033894 Accuracy: 99.06% 17 Validation loss: 0.056981 Best loss: 0.033894 Accuracy: 98.87% 18 Validation loss: 0.053657 Best loss: 0.033894 Accuracy: 98.87% 19 Validation loss: 0.038305 Best loss: 0.033894 Accuracy: 99.18% 20 Validation loss: 0.058509 Best loss: 0.033894 Accuracy: 98.98% 21 Validation loss: 0.068324 Best loss: 0.033894 Accuracy: 98.63% 22 Validation loss: 0.039071 Best loss: 0.033894 Accuracy: 99.14% 23 Validation loss: 0.047983 Best loss: 0.033894 Accuracy: 99.14% 24 Validation loss: 0.045884 Best loss: 0.033894 Accuracy: 99.10% 25 Validation loss: 0.043597 Best loss: 0.033894 Accuracy: 98.91% 26 Validation loss: 0.036006 Best loss: 0.033894 Accuracy: 99.14% 27 Validation loss: 0.055784 Best loss: 0.033894 Accuracy: 98.94% 28 Validation loss: 0.055750 Best loss: 0.033894 Accuracy: 99.02% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 42.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.107053 Best loss: 0.107053 Accuracy: 98.12% 1 Validation loss: 0.073647 Best loss: 0.073647 Accuracy: 98.08% 2 Validation loss: 0.150474 Best loss: 0.073647 Accuracy: 95.47% 3 Validation loss: 0.043393 Best loss: 0.043393 Accuracy: 98.83% 4 Validation loss: 0.058746 Best loss: 0.043393 Accuracy: 99.02% 5 Validation loss: 0.031025 Best loss: 0.031025 Accuracy: 98.91% 6 Validation loss: 0.061243 Best loss: 0.031025 Accuracy: 98.63% 7 Validation loss: 0.072707 Best loss: 0.031025 Accuracy: 98.24% 8 Validation loss: 0.029278 Best loss: 0.029278 Accuracy: 99.18% 9 Validation loss: 0.045804 Best loss: 0.029278 Accuracy: 98.94% 10 Validation loss: 0.047857 Best loss: 0.029278 Accuracy: 99.06% 11 Validation loss: 0.056693 Best loss: 0.029278 Accuracy: 98.94% 12 Validation loss: 0.049525 Best loss: 0.029278 Accuracy: 99.02% 13 Validation loss: 0.044987 Best loss: 0.029278 Accuracy: 98.83% 14 Validation loss: 0.078204 Best loss: 0.029278 Accuracy: 98.44% 15 Validation loss: 0.058207 Best loss: 0.029278 Accuracy: 98.87% 16 Validation loss: 0.060972 Best loss: 0.029278 Accuracy: 98.55% 17 Validation loss: 0.048072 Best loss: 0.029278 Accuracy: 99.06% 18 Validation loss: 0.042469 Best loss: 0.029278 Accuracy: 98.91% 19 Validation loss: 0.044807 Best loss: 0.029278 Accuracy: 98.83% 20 Validation loss: 0.045519 Best loss: 0.029278 Accuracy: 99.02% 21 Validation loss: 0.047861 Best loss: 0.029278 Accuracy: 98.87% 22 Validation loss: 0.060079 Best loss: 0.029278 Accuracy: 98.55% 23 Validation loss: 0.060600 Best loss: 0.029278 Accuracy: 98.94% 24 Validation loss: 0.029064 Best loss: 0.029064 Accuracy: 99.22% 25 Validation loss: 0.055717 Best loss: 0.029064 Accuracy: 98.40% 26 Validation loss: 0.041384 Best loss: 0.029064 Accuracy: 98.94% 27 Validation loss: 0.069328 Best loss: 0.029064 Accuracy: 98.12% 28 Validation loss: 0.049314 Best loss: 0.029064 Accuracy: 99.18% 29 Validation loss: 0.057784 Best loss: 0.029064 Accuracy: 98.87% 30 Validation loss: 0.029942 Best loss: 0.029064 Accuracy: 99.34% 31 Validation loss: 0.038150 Best loss: 0.029064 Accuracy: 99.22% 32 Validation loss: 0.027812 Best loss: 0.027812 Accuracy: 99.26% 33 Validation loss: 0.042989 Best loss: 0.027812 Accuracy: 99.10% 34 Validation loss: 0.064543 Best loss: 0.027812 Accuracy: 98.71% 35 Validation loss: 0.044508 Best loss: 0.027812 Accuracy: 99.34% 36 Validation loss: 0.055670 Best loss: 0.027812 Accuracy: 98.91% 37 Validation loss: 0.037337 Best loss: 0.027812 Accuracy: 99.10% 38 Validation loss: 0.040792 Best loss: 0.027812 Accuracy: 98.98% 39 Validation loss: 0.039906 Best loss: 0.027812 Accuracy: 98.98% 40 Validation loss: 0.057500 Best loss: 0.027812 Accuracy: 98.87% 41 Validation loss: 0.033938 Best loss: 0.027812 Accuracy: 99.34% 42 Validation loss: 0.028627 Best loss: 0.027812 Accuracy: 99.34% 43 Validation loss: 0.047478 Best loss: 0.027812 Accuracy: 99.06% 44 Validation loss: 0.032758 Best loss: 0.027812 Accuracy: 99.41% 45 Validation loss: 0.054389 Best loss: 0.027812 Accuracy: 98.98% 46 Validation loss: 0.055367 Best loss: 0.027812 Accuracy: 98.83% 47 Validation loss: 0.040697 Best loss: 0.027812 Accuracy: 99.26% 48 Validation loss: 0.036535 Best loss: 0.027812 Accuracy: 99.22% 49 Validation loss: 0.041116 Best loss: 0.027812 Accuracy: 99.10% 50 Validation loss: 0.035240 Best loss: 0.027812 Accuracy: 99.41% 51 Validation loss: 0.046781 Best loss: 0.027812 Accuracy: 99.26% 52 Validation loss: 0.040183 Best loss: 0.027812 Accuracy: 99.22% 53 Validation loss: 0.045523 Best loss: 0.027812 Accuracy: 99.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=160, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.184481 Best loss: 0.184481 Accuracy: 94.33% 1 Validation loss: 0.102338 Best loss: 0.102338 Accuracy: 96.56% 2 Validation loss: 0.093564 Best loss: 0.093564 Accuracy: 97.22% 3 Validation loss: 0.093499 Best loss: 0.093499 Accuracy: 96.95% 4 Validation loss: 0.065669 Best loss: 0.065669 Accuracy: 97.81% 5 Validation loss: 0.087710 Best loss: 0.065669 Accuracy: 97.46% 6 Validation loss: 0.055780 Best loss: 0.055780 Accuracy: 98.24% 7 Validation loss: 0.084153 Best loss: 0.055780 Accuracy: 97.93% 8 Validation loss: 0.063375 Best loss: 0.055780 Accuracy: 98.12% 9 Validation loss: 0.060442 Best loss: 0.055780 Accuracy: 98.24% 10 Validation loss: 0.066962 Best loss: 0.055780 Accuracy: 98.28% 11 Validation loss: 0.057897 Best loss: 0.055780 Accuracy: 98.36% 12 Validation loss: 0.074409 Best loss: 0.055780 Accuracy: 97.93% 13 Validation loss: 0.089372 Best loss: 0.055780 Accuracy: 97.77% 14 Validation loss: 0.073348 Best loss: 0.055780 Accuracy: 98.16% 15 Validation loss: 0.065285 Best loss: 0.055780 Accuracy: 98.16% 16 Validation loss: 0.074332 Best loss: 0.055780 Accuracy: 98.28% 17 Validation loss: 0.068792 Best loss: 0.055780 Accuracy: 98.63% 18 Validation loss: 0.072621 Best loss: 0.055780 Accuracy: 98.40% 19 Validation loss: 0.057160 Best loss: 0.055780 Accuracy: 98.63% 20 Validation loss: 0.057101 Best loss: 0.055780 Accuracy: 98.75% 21 Validation loss: 0.068835 Best loss: 0.055780 Accuracy: 98.55% 22 Validation loss: 0.085497 Best loss: 0.055780 Accuracy: 98.32% 23 Validation loss: 0.083567 Best loss: 0.055780 Accuracy: 97.89% 24 Validation loss: 0.054635 Best loss: 0.054635 Accuracy: 98.63% 25 Validation loss: 0.078551 Best loss: 0.054635 Accuracy: 98.24% 26 Validation loss: 0.075033 Best loss: 0.054635 Accuracy: 98.44% 27 Validation loss: 0.077562 Best loss: 0.054635 Accuracy: 98.67% 28 Validation loss: 0.074838 Best loss: 0.054635 Accuracy: 98.48% 29 Validation loss: 0.082496 Best loss: 0.054635 Accuracy: 98.12% 30 Validation loss: 0.067076 Best loss: 0.054635 Accuracy: 98.59% 31 Validation loss: 0.065916 Best loss: 0.054635 Accuracy: 98.79% 32 Validation loss: 0.079969 Best loss: 0.054635 Accuracy: 98.51% 33 Validation loss: 0.078265 Best loss: 0.054635 Accuracy: 98.36% 34 Validation loss: 0.074596 Best loss: 0.054635 Accuracy: 98.32% 35 Validation loss: 0.073940 Best loss: 0.054635 Accuracy: 98.59% 36 Validation loss: 0.066097 Best loss: 0.054635 Accuracy: 98.79% 37 Validation loss: 0.062299 Best loss: 0.054635 Accuracy: 98.79% 38 Validation loss: 0.091091 Best loss: 0.054635 Accuracy: 98.40% 39 Validation loss: 0.109991 Best loss: 0.054635 Accuracy: 98.32% 40 Validation loss: 0.067946 Best loss: 0.054635 Accuracy: 98.59% 41 Validation loss: 0.076488 Best loss: 0.054635 Accuracy: 98.55% 42 Validation loss: 0.072383 Best loss: 0.054635 Accuracy: 98.71% 43 Validation loss: 0.087592 Best loss: 0.054635 Accuracy: 98.40% 44 Validation loss: 0.062474 Best loss: 0.054635 Accuracy: 98.91% 45 Validation loss: 0.078275 Best loss: 0.054635 Accuracy: 98.40% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 16.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.160814 Best loss: 0.160814 Accuracy: 95.31% 1 Validation loss: 0.102249 Best loss: 0.102249 Accuracy: 96.79% 2 Validation loss: 0.079475 Best loss: 0.079475 Accuracy: 97.26% 3 Validation loss: 0.077620 Best loss: 0.077620 Accuracy: 97.77% 4 Validation loss: 0.067822 Best loss: 0.067822 Accuracy: 98.05% 5 Validation loss: 0.064370 Best loss: 0.064370 Accuracy: 97.93% 6 Validation loss: 0.056286 Best loss: 0.056286 Accuracy: 97.85% 7 Validation loss: 0.073493 Best loss: 0.056286 Accuracy: 97.89% 8 Validation loss: 0.055696 Best loss: 0.055696 Accuracy: 98.24% 9 Validation loss: 0.063300 Best loss: 0.055696 Accuracy: 98.12% 10 Validation loss: 0.054845 Best loss: 0.054845 Accuracy: 98.36% 11 Validation loss: 0.066661 Best loss: 0.054845 Accuracy: 98.20% 12 Validation loss: 0.065943 Best loss: 0.054845 Accuracy: 98.36% 13 Validation loss: 0.058053 Best loss: 0.054845 Accuracy: 98.32% 14 Validation loss: 0.071458 Best loss: 0.054845 Accuracy: 98.40% 15 Validation loss: 0.067906 Best loss: 0.054845 Accuracy: 98.40% 16 Validation loss: 0.064611 Best loss: 0.054845 Accuracy: 98.36% 17 Validation loss: 0.054438 Best loss: 0.054438 Accuracy: 98.75% 18 Validation loss: 0.046883 Best loss: 0.046883 Accuracy: 98.87% 19 Validation loss: 0.055761 Best loss: 0.046883 Accuracy: 98.63% 20 Validation loss: 0.063001 Best loss: 0.046883 Accuracy: 98.59% 21 Validation loss: 0.075577 Best loss: 0.046883 Accuracy: 98.24% 22 Validation loss: 0.071873 Best loss: 0.046883 Accuracy: 98.32% 23 Validation loss: 0.047135 Best loss: 0.046883 Accuracy: 98.71% 24 Validation loss: 0.058850 Best loss: 0.046883 Accuracy: 98.51% 25 Validation loss: 0.093504 Best loss: 0.046883 Accuracy: 98.32% 26 Validation loss: 0.054652 Best loss: 0.046883 Accuracy: 98.94% 27 Validation loss: 0.054776 Best loss: 0.046883 Accuracy: 98.59% 28 Validation loss: 0.066783 Best loss: 0.046883 Accuracy: 98.55% 29 Validation loss: 0.043654 Best loss: 0.043654 Accuracy: 99.14% 30 Validation loss: 0.051505 Best loss: 0.043654 Accuracy: 98.98% 31 Validation loss: 0.047586 Best loss: 0.043654 Accuracy: 98.98% 32 Validation loss: 0.047705 Best loss: 0.043654 Accuracy: 98.94% 33 Validation loss: 0.054532 Best loss: 0.043654 Accuracy: 99.10% 34 Validation loss: 0.046501 Best loss: 0.043654 Accuracy: 98.87% 35 Validation loss: 0.078441 Best loss: 0.043654 Accuracy: 98.48% 36 Validation loss: 0.069256 Best loss: 0.043654 Accuracy: 98.51% 37 Validation loss: 0.062751 Best loss: 0.043654 Accuracy: 98.71% 38 Validation loss: 0.076100 Best loss: 0.043654 Accuracy: 98.63% 39 Validation loss: 0.068529 Best loss: 0.043654 Accuracy: 98.71% 40 Validation loss: 0.072260 Best loss: 0.043654 Accuracy: 98.48% 41 Validation loss: 0.069181 Best loss: 0.043654 Accuracy: 98.51% 42 Validation loss: 0.060156 Best loss: 0.043654 Accuracy: 98.87% 43 Validation loss: 0.062395 Best loss: 0.043654 Accuracy: 98.71% 44 Validation loss: 0.054642 Best loss: 0.043654 Accuracy: 98.91% 45 Validation loss: 0.057578 Best loss: 0.043654 Accuracy: 98.83% 46 Validation loss: 0.066789 Best loss: 0.043654 Accuracy: 98.91% 47 Validation loss: 0.061094 Best loss: 0.043654 Accuracy: 98.98% 48 Validation loss: 0.059561 Best loss: 0.043654 Accuracy: 98.71% 49 Validation loss: 0.064222 Best loss: 0.043654 Accuracy: 98.79% 50 Validation loss: 0.070753 Best loss: 0.043654 Accuracy: 98.59% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 17.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.158781 Best loss: 0.158781 Accuracy: 95.39% 1 Validation loss: 0.095630 Best loss: 0.095630 Accuracy: 97.30% 2 Validation loss: 0.079126 Best loss: 0.079126 Accuracy: 97.38% 3 Validation loss: 0.068453 Best loss: 0.068453 Accuracy: 97.89% 4 Validation loss: 0.090842 Best loss: 0.068453 Accuracy: 97.26% 5 Validation loss: 0.086395 Best loss: 0.068453 Accuracy: 97.54% 6 Validation loss: 0.070124 Best loss: 0.068453 Accuracy: 97.69% 7 Validation loss: 0.075179 Best loss: 0.068453 Accuracy: 97.69% 8 Validation loss: 0.076768 Best loss: 0.068453 Accuracy: 98.12% 9 Validation loss: 0.065066 Best loss: 0.065066 Accuracy: 98.36% 10 Validation loss: 0.066836 Best loss: 0.065066 Accuracy: 98.20% 11 Validation loss: 0.075382 Best loss: 0.065066 Accuracy: 98.12% 12 Validation loss: 0.061539 Best loss: 0.061539 Accuracy: 98.51% 13 Validation loss: 0.092606 Best loss: 0.061539 Accuracy: 98.01% 14 Validation loss: 0.068043 Best loss: 0.061539 Accuracy: 98.44% 15 Validation loss: 0.063387 Best loss: 0.061539 Accuracy: 98.48% 16 Validation loss: 0.057121 Best loss: 0.057121 Accuracy: 98.67% 17 Validation loss: 0.072127 Best loss: 0.057121 Accuracy: 98.36% 18 Validation loss: 0.067132 Best loss: 0.057121 Accuracy: 98.44% 19 Validation loss: 0.071671 Best loss: 0.057121 Accuracy: 98.48% 20 Validation loss: 0.075482 Best loss: 0.057121 Accuracy: 98.83% 21 Validation loss: 0.084640 Best loss: 0.057121 Accuracy: 98.59% 22 Validation loss: 0.074494 Best loss: 0.057121 Accuracy: 98.55% 23 Validation loss: 0.075931 Best loss: 0.057121 Accuracy: 98.40% 24 Validation loss: 0.069990 Best loss: 0.057121 Accuracy: 98.79% 25 Validation loss: 0.068442 Best loss: 0.057121 Accuracy: 98.63% 26 Validation loss: 0.076991 Best loss: 0.057121 Accuracy: 98.51% 27 Validation loss: 0.066098 Best loss: 0.057121 Accuracy: 98.48% 28 Validation loss: 0.072465 Best loss: 0.057121 Accuracy: 98.63% 29 Validation loss: 0.084986 Best loss: 0.057121 Accuracy: 98.67% 30 Validation loss: 0.095733 Best loss: 0.057121 Accuracy: 98.63% 31 Validation loss: 0.077313 Best loss: 0.057121 Accuracy: 98.75% 32 Validation loss: 0.091947 Best loss: 0.057121 Accuracy: 98.75% 33 Validation loss: 0.096341 Best loss: 0.057121 Accuracy: 98.36% 34 Validation loss: 0.096775 Best loss: 0.057121 Accuracy: 98.48% 35 Validation loss: 0.113042 Best loss: 0.057121 Accuracy: 98.32% 36 Validation loss: 0.082718 Best loss: 0.057121 Accuracy: 98.67% 37 Validation loss: 0.082157 Best loss: 0.057121 Accuracy: 98.51% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 13.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.182367 Best loss: 0.182367 Accuracy: 95.58% 1 Validation loss: 0.085656 Best loss: 0.085656 Accuracy: 97.26% 2 Validation loss: 0.070544 Best loss: 0.070544 Accuracy: 97.69% 3 Validation loss: 0.070267 Best loss: 0.070267 Accuracy: 97.69% 4 Validation loss: 0.058181 Best loss: 0.058181 Accuracy: 98.08% 5 Validation loss: 0.078422 Best loss: 0.058181 Accuracy: 97.58% 6 Validation loss: 0.062637 Best loss: 0.058181 Accuracy: 98.12% 7 Validation loss: 0.072922 Best loss: 0.058181 Accuracy: 97.89% 8 Validation loss: 0.071414 Best loss: 0.058181 Accuracy: 98.28% 9 Validation loss: 0.057423 Best loss: 0.057423 Accuracy: 98.59% 10 Validation loss: 0.055427 Best loss: 0.055427 Accuracy: 98.75% 11 Validation loss: 0.064498 Best loss: 0.055427 Accuracy: 98.40% 12 Validation loss: 0.054970 Best loss: 0.054970 Accuracy: 98.83% 13 Validation loss: 0.057596 Best loss: 0.054970 Accuracy: 98.48% 14 Validation loss: 0.061400 Best loss: 0.054970 Accuracy: 98.55% 15 Validation loss: 0.073960 Best loss: 0.054970 Accuracy: 98.32% 16 Validation loss: 0.054029 Best loss: 0.054029 Accuracy: 98.79% 17 Validation loss: 0.059498 Best loss: 0.054029 Accuracy: 98.91% 18 Validation loss: 0.073712 Best loss: 0.054029 Accuracy: 98.59% 19 Validation loss: 0.067785 Best loss: 0.054029 Accuracy: 98.75% 20 Validation loss: 0.097018 Best loss: 0.054029 Accuracy: 97.81% 21 Validation loss: 0.062310 Best loss: 0.054029 Accuracy: 98.51% 22 Validation loss: 0.067121 Best loss: 0.054029 Accuracy: 98.71% 23 Validation loss: 0.068144 Best loss: 0.054029 Accuracy: 98.71% 24 Validation loss: 0.074855 Best loss: 0.054029 Accuracy: 98.51% 25 Validation loss: 0.056829 Best loss: 0.054029 Accuracy: 98.75% 26 Validation loss: 0.053958 Best loss: 0.053958 Accuracy: 98.67% 27 Validation loss: 0.061520 Best loss: 0.053958 Accuracy: 98.83% 28 Validation loss: 0.061340 Best loss: 0.053958 Accuracy: 99.06% 29 Validation loss: 0.076034 Best loss: 0.053958 Accuracy: 98.44% 30 Validation loss: 0.075102 Best loss: 0.053958 Accuracy: 98.59% 31 Validation loss: 0.069385 Best loss: 0.053958 Accuracy: 98.67% 32 Validation loss: 0.083336 Best loss: 0.053958 Accuracy: 98.40% 33 Validation loss: 0.149369 Best loss: 0.053958 Accuracy: 97.30% 34 Validation loss: 0.072930 Best loss: 0.053958 Accuracy: 98.71% 35 Validation loss: 0.074795 Best loss: 0.053958 Accuracy: 98.71% 36 Validation loss: 0.075284 Best loss: 0.053958 Accuracy: 98.75% 37 Validation loss: 0.087039 Best loss: 0.053958 Accuracy: 98.48% 38 Validation loss: 0.077371 Best loss: 0.053958 Accuracy: 98.67% 39 Validation loss: 0.101866 Best loss: 0.053958 Accuracy: 98.48% 40 Validation loss: 0.069555 Best loss: 0.053958 Accuracy: 98.59% 41 Validation loss: 0.080147 Best loss: 0.053958 Accuracy: 98.63% 42 Validation loss: 0.071321 Best loss: 0.053958 Accuracy: 99.10% 43 Validation loss: 0.071368 Best loss: 0.053958 Accuracy: 98.63% 44 Validation loss: 0.068690 Best loss: 0.053958 Accuracy: 98.48% 45 Validation loss: 0.057069 Best loss: 0.053958 Accuracy: 98.75% 46 Validation loss: 0.065785 Best loss: 0.053958 Accuracy: 98.87% 47 Validation loss: 0.054369 Best loss: 0.053958 Accuracy: 99.02% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 16.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.153168 Best loss: 0.153168 Accuracy: 95.82% 1 Validation loss: 0.073450 Best loss: 0.073450 Accuracy: 97.73% 2 Validation loss: 0.061361 Best loss: 0.061361 Accuracy: 97.89% 3 Validation loss: 0.055724 Best loss: 0.055724 Accuracy: 98.32% 4 Validation loss: 0.065650 Best loss: 0.055724 Accuracy: 98.12% 5 Validation loss: 0.059432 Best loss: 0.055724 Accuracy: 98.20% 6 Validation loss: 0.053373 Best loss: 0.053373 Accuracy: 98.44% 7 Validation loss: 0.058979 Best loss: 0.053373 Accuracy: 98.59% 8 Validation loss: 0.063963 Best loss: 0.053373 Accuracy: 98.40% 9 Validation loss: 0.068495 Best loss: 0.053373 Accuracy: 98.32% 10 Validation loss: 0.060203 Best loss: 0.053373 Accuracy: 98.71% 11 Validation loss: 0.054139 Best loss: 0.053373 Accuracy: 98.59% 12 Validation loss: 0.059998 Best loss: 0.053373 Accuracy: 98.48% 13 Validation loss: 0.047157 Best loss: 0.047157 Accuracy: 98.79% 14 Validation loss: 0.060287 Best loss: 0.047157 Accuracy: 98.63% 15 Validation loss: 0.072705 Best loss: 0.047157 Accuracy: 98.40% 16 Validation loss: 0.055444 Best loss: 0.047157 Accuracy: 98.63% 17 Validation loss: 0.051747 Best loss: 0.047157 Accuracy: 98.91% 18 Validation loss: 0.047786 Best loss: 0.047157 Accuracy: 98.94% 19 Validation loss: 0.061242 Best loss: 0.047157 Accuracy: 98.67% 20 Validation loss: 0.078552 Best loss: 0.047157 Accuracy: 98.63% 21 Validation loss: 0.075887 Best loss: 0.047157 Accuracy: 98.01% 22 Validation loss: 0.064220 Best loss: 0.047157 Accuracy: 98.55% 23 Validation loss: 0.053029 Best loss: 0.047157 Accuracy: 99.02% 24 Validation loss: 0.062262 Best loss: 0.047157 Accuracy: 98.55% 25 Validation loss: 0.054432 Best loss: 0.047157 Accuracy: 98.75% 26 Validation loss: 0.056494 Best loss: 0.047157 Accuracy: 98.87% 27 Validation loss: 0.060465 Best loss: 0.047157 Accuracy: 98.71% 28 Validation loss: 0.053346 Best loss: 0.047157 Accuracy: 98.91% 29 Validation loss: 0.065667 Best loss: 0.047157 Accuracy: 98.71% 30 Validation loss: 0.052683 Best loss: 0.047157 Accuracy: 98.79% 31 Validation loss: 0.055981 Best loss: 0.047157 Accuracy: 98.87% 32 Validation loss: 0.055771 Best loss: 0.047157 Accuracy: 98.83% 33 Validation loss: 0.070004 Best loss: 0.047157 Accuracy: 98.59% 34 Validation loss: 0.059212 Best loss: 0.047157 Accuracy: 98.83% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 13.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.130347 Best loss: 0.130347 Accuracy: 96.48% 1 Validation loss: 0.079729 Best loss: 0.079729 Accuracy: 97.30% 2 Validation loss: 0.056235 Best loss: 0.056235 Accuracy: 98.16% 3 Validation loss: 0.065111 Best loss: 0.056235 Accuracy: 98.05% 4 Validation loss: 0.061431 Best loss: 0.056235 Accuracy: 98.12% 5 Validation loss: 0.053882 Best loss: 0.053882 Accuracy: 98.40% 6 Validation loss: 0.043720 Best loss: 0.043720 Accuracy: 98.67% 7 Validation loss: 0.054346 Best loss: 0.043720 Accuracy: 98.16% 8 Validation loss: 0.050126 Best loss: 0.043720 Accuracy: 98.55% 9 Validation loss: 0.055758 Best loss: 0.043720 Accuracy: 98.55% 10 Validation loss: 0.056214 Best loss: 0.043720 Accuracy: 98.67% 11 Validation loss: 0.066883 Best loss: 0.043720 Accuracy: 98.20% 12 Validation loss: 0.050982 Best loss: 0.043720 Accuracy: 98.71% 13 Validation loss: 0.057517 Best loss: 0.043720 Accuracy: 98.75% 14 Validation loss: 0.054107 Best loss: 0.043720 Accuracy: 98.71% 15 Validation loss: 0.054419 Best loss: 0.043720 Accuracy: 98.71% 16 Validation loss: 0.061426 Best loss: 0.043720 Accuracy: 98.83% 17 Validation loss: 0.052896 Best loss: 0.043720 Accuracy: 98.83% 18 Validation loss: 0.056873 Best loss: 0.043720 Accuracy: 98.79% 19 Validation loss: 0.066514 Best loss: 0.043720 Accuracy: 98.51% 20 Validation loss: 0.051471 Best loss: 0.043720 Accuracy: 98.75% 21 Validation loss: 0.062363 Best loss: 0.043720 Accuracy: 98.28% 22 Validation loss: 0.057841 Best loss: 0.043720 Accuracy: 98.67% 23 Validation loss: 0.063764 Best loss: 0.043720 Accuracy: 98.67% 24 Validation loss: 0.056741 Best loss: 0.043720 Accuracy: 98.91% 25 Validation loss: 0.076910 Best loss: 0.043720 Accuracy: 98.59% 26 Validation loss: 0.054848 Best loss: 0.043720 Accuracy: 98.79% 27 Validation loss: 0.062068 Best loss: 0.043720 Accuracy: 98.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=90, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.1, total= 11.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.059186 Best loss: 0.059186 Accuracy: 98.20% 1 Validation loss: 0.054346 Best loss: 0.054346 Accuracy: 98.36% 2 Validation loss: 0.056004 Best loss: 0.054346 Accuracy: 98.40% 3 Validation loss: 0.050929 Best loss: 0.050929 Accuracy: 98.48% 4 Validation loss: 0.044774 Best loss: 0.044774 Accuracy: 98.55% 5 Validation loss: 0.063533 Best loss: 0.044774 Accuracy: 98.20% 6 Validation loss: 0.046824 Best loss: 0.044774 Accuracy: 98.55% 7 Validation loss: 0.053055 Best loss: 0.044774 Accuracy: 98.59% 8 Validation loss: 0.056151 Best loss: 0.044774 Accuracy: 98.59% 9 Validation loss: 0.058731 Best loss: 0.044774 Accuracy: 98.63% 10 Validation loss: 0.056158 Best loss: 0.044774 Accuracy: 98.67% 11 Validation loss: 0.051780 Best loss: 0.044774 Accuracy: 98.98% 12 Validation loss: 0.071254 Best loss: 0.044774 Accuracy: 98.59% 13 Validation loss: 0.090287 Best loss: 0.044774 Accuracy: 98.01% 14 Validation loss: 0.063131 Best loss: 0.044774 Accuracy: 98.51% 15 Validation loss: 0.054605 Best loss: 0.044774 Accuracy: 98.67% 16 Validation loss: 0.051012 Best loss: 0.044774 Accuracy: 98.48% 17 Validation loss: 0.051090 Best loss: 0.044774 Accuracy: 98.87% 18 Validation loss: 0.060417 Best loss: 0.044774 Accuracy: 98.94% 19 Validation loss: 0.052143 Best loss: 0.044774 Accuracy: 99.14% 20 Validation loss: 0.051652 Best loss: 0.044774 Accuracy: 99.06% 21 Validation loss: 0.064041 Best loss: 0.044774 Accuracy: 98.83% 22 Validation loss: 0.061378 Best loss: 0.044774 Accuracy: 98.71% 23 Validation loss: 0.061061 Best loss: 0.044774 Accuracy: 98.51% 24 Validation loss: 0.056880 Best loss: 0.044774 Accuracy: 98.94% 25 Validation loss: 0.067641 Best loss: 0.044774 Accuracy: 98.71% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 9.0s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.067021 Best loss: 0.067021 Accuracy: 98.16% 1 Validation loss: 0.049104 Best loss: 0.049104 Accuracy: 98.55% 2 Validation loss: 0.043754 Best loss: 0.043754 Accuracy: 98.59% 3 Validation loss: 0.038169 Best loss: 0.038169 Accuracy: 98.67% 4 Validation loss: 0.052283 Best loss: 0.038169 Accuracy: 98.48% 5 Validation loss: 0.051412 Best loss: 0.038169 Accuracy: 98.71% 6 Validation loss: 0.047881 Best loss: 0.038169 Accuracy: 98.75% 7 Validation loss: 0.044049 Best loss: 0.038169 Accuracy: 98.94% 8 Validation loss: 0.046057 Best loss: 0.038169 Accuracy: 98.59% 9 Validation loss: 0.068976 Best loss: 0.038169 Accuracy: 98.55% 10 Validation loss: 0.065399 Best loss: 0.038169 Accuracy: 98.55% 11 Validation loss: 0.059398 Best loss: 0.038169 Accuracy: 98.59% 12 Validation loss: 0.041762 Best loss: 0.038169 Accuracy: 98.98% 13 Validation loss: 0.047668 Best loss: 0.038169 Accuracy: 99.14% 14 Validation loss: 0.046381 Best loss: 0.038169 Accuracy: 99.02% 15 Validation loss: 0.053880 Best loss: 0.038169 Accuracy: 98.94% 16 Validation loss: 0.058541 Best loss: 0.038169 Accuracy: 98.87% 17 Validation loss: 0.058276 Best loss: 0.038169 Accuracy: 98.51% 18 Validation loss: 0.042211 Best loss: 0.038169 Accuracy: 98.98% 19 Validation loss: 0.035846 Best loss: 0.035846 Accuracy: 99.06% 20 Validation loss: 0.042894 Best loss: 0.035846 Accuracy: 98.98% 21 Validation loss: 0.051795 Best loss: 0.035846 Accuracy: 98.98% 22 Validation loss: 0.064241 Best loss: 0.035846 Accuracy: 98.83% 23 Validation loss: 0.059420 Best loss: 0.035846 Accuracy: 98.71% 24 Validation loss: 0.057099 Best loss: 0.035846 Accuracy: 98.91% 25 Validation loss: 0.058248 Best loss: 0.035846 Accuracy: 98.63% 26 Validation loss: 0.050468 Best loss: 0.035846 Accuracy: 98.91% 27 Validation loss: 0.067709 Best loss: 0.035846 Accuracy: 98.36% 28 Validation loss: 0.051668 Best loss: 0.035846 Accuracy: 98.94% 29 Validation loss: 0.045849 Best loss: 0.035846 Accuracy: 98.91% 30 Validation loss: 0.047275 Best loss: 0.035846 Accuracy: 98.94% 31 Validation loss: 0.041406 Best loss: 0.035846 Accuracy: 98.87% 32 Validation loss: 0.046680 Best loss: 0.035846 Accuracy: 99.06% 33 Validation loss: 0.046171 Best loss: 0.035846 Accuracy: 99.06% 34 Validation loss: 0.065107 Best loss: 0.035846 Accuracy: 98.79% 35 Validation loss: 0.048322 Best loss: 0.035846 Accuracy: 98.91% 36 Validation loss: 0.040941 Best loss: 0.035846 Accuracy: 99.06% 37 Validation loss: 0.045992 Best loss: 0.035846 Accuracy: 98.91% 38 Validation loss: 0.043259 Best loss: 0.035846 Accuracy: 99.02% 39 Validation loss: 0.042103 Best loss: 0.035846 Accuracy: 99.10% 40 Validation loss: 0.042371 Best loss: 0.035846 Accuracy: 99.10% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 13.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02 0 Validation loss: 0.052266 Best loss: 0.052266 Accuracy: 98.51% 1 Validation loss: 0.058842 Best loss: 0.052266 Accuracy: 98.32% 2 Validation loss: 0.047205 Best loss: 0.047205 Accuracy: 98.24% 3 Validation loss: 0.041995 Best loss: 0.041995 Accuracy: 98.63% 4 Validation loss: 0.033776 Best loss: 0.033776 Accuracy: 99.02% 5 Validation loss: 0.065838 Best loss: 0.033776 Accuracy: 98.20% 6 Validation loss: 0.043951 Best loss: 0.033776 Accuracy: 98.55% 7 Validation loss: 0.050212 Best loss: 0.033776 Accuracy: 98.59% 8 Validation loss: 0.054884 Best loss: 0.033776 Accuracy: 98.44% 9 Validation loss: 0.037355 Best loss: 0.033776 Accuracy: 99.10% 10 Validation loss: 0.063130 Best loss: 0.033776 Accuracy: 98.67% 11 Validation loss: 0.067459 Best loss: 0.033776 Accuracy: 98.59% 12 Validation loss: 0.043296 Best loss: 0.033776 Accuracy: 98.98% 13 Validation loss: 0.045487 Best loss: 0.033776 Accuracy: 99.02% 14 Validation loss: 0.055749 Best loss: 0.033776 Accuracy: 98.75% 15 Validation loss: 0.047854 Best loss: 0.033776 Accuracy: 98.87% 16 Validation loss: 0.043328 Best loss: 0.033776 Accuracy: 99.06% 17 Validation loss: 0.045548 Best loss: 0.033776 Accuracy: 99.02% 18 Validation loss: 0.037475 Best loss: 0.033776 Accuracy: 99.10% 19 Validation loss: 0.036969 Best loss: 0.033776 Accuracy: 98.98% 20 Validation loss: 0.057609 Best loss: 0.033776 Accuracy: 98.98% 21 Validation loss: 0.047549 Best loss: 0.033776 Accuracy: 99.02% 22 Validation loss: 0.050357 Best loss: 0.033776 Accuracy: 98.67% 23 Validation loss: 0.049700 Best loss: 0.033776 Accuracy: 98.91% 24 Validation loss: 0.044264 Best loss: 0.033776 Accuracy: 99.06% 25 Validation loss: 0.034310 Best loss: 0.033776 Accuracy: 99.30% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.02, total= 9.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.067335 Best loss: 0.067335 Accuracy: 97.97% 1 Validation loss: 0.062334 Best loss: 0.062334 Accuracy: 97.93% 2 Validation loss: 0.053975 Best loss: 0.053975 Accuracy: 98.32% 3 Validation loss: 0.046190 Best loss: 0.046190 Accuracy: 98.63% 4 Validation loss: 0.034669 Best loss: 0.034669 Accuracy: 99.02% 5 Validation loss: 0.047362 Best loss: 0.034669 Accuracy: 98.67% 6 Validation loss: 0.043436 Best loss: 0.034669 Accuracy: 98.67% 7 Validation loss: 0.041201 Best loss: 0.034669 Accuracy: 99.02% 8 Validation loss: 0.043692 Best loss: 0.034669 Accuracy: 98.87% 9 Validation loss: 0.055365 Best loss: 0.034669 Accuracy: 98.51% 10 Validation loss: 0.066345 Best loss: 0.034669 Accuracy: 98.40% 11 Validation loss: 0.057643 Best loss: 0.034669 Accuracy: 98.75% 12 Validation loss: 0.047792 Best loss: 0.034669 Accuracy: 98.87% 13 Validation loss: 0.045471 Best loss: 0.034669 Accuracy: 98.94% 14 Validation loss: 0.040026 Best loss: 0.034669 Accuracy: 98.98% 15 Validation loss: 0.049226 Best loss: 0.034669 Accuracy: 98.94% 16 Validation loss: 0.042457 Best loss: 0.034669 Accuracy: 99.06% 17 Validation loss: 0.050335 Best loss: 0.034669 Accuracy: 98.98% 18 Validation loss: 0.056135 Best loss: 0.034669 Accuracy: 98.79% 19 Validation loss: 0.057876 Best loss: 0.034669 Accuracy: 98.79% 20 Validation loss: 0.050595 Best loss: 0.034669 Accuracy: 98.91% 21 Validation loss: 0.042160 Best loss: 0.034669 Accuracy: 98.79% 22 Validation loss: 0.041114 Best loss: 0.034669 Accuracy: 99.14% 23 Validation loss: 0.045735 Best loss: 0.034669 Accuracy: 98.91% 24 Validation loss: 0.050992 Best loss: 0.034669 Accuracy: 98.83% 25 Validation loss: 0.050492 Best loss: 0.034669 Accuracy: 98.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01, total= 10.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.060937 Best loss: 0.060937 Accuracy: 98.20% 1 Validation loss: 0.044441 Best loss: 0.044441 Accuracy: 98.44% 2 Validation loss: 0.046397 Best loss: 0.044441 Accuracy: 98.55% 3 Validation loss: 0.040401 Best loss: 0.040401 Accuracy: 98.71% 4 Validation loss: 0.050154 Best loss: 0.040401 Accuracy: 98.67% 5 Validation loss: 0.050609 Best loss: 0.040401 Accuracy: 98.67% 6 Validation loss: 0.049682 Best loss: 0.040401 Accuracy: 98.63% 7 Validation loss: 0.049771 Best loss: 0.040401 Accuracy: 98.71% 8 Validation loss: 0.048441 Best loss: 0.040401 Accuracy: 98.63% 9 Validation loss: 0.052021 Best loss: 0.040401 Accuracy: 98.87% 10 Validation loss: 0.048821 Best loss: 0.040401 Accuracy: 98.94% 11 Validation loss: 0.056838 Best loss: 0.040401 Accuracy: 98.87% 12 Validation loss: 0.035479 Best loss: 0.035479 Accuracy: 99.10% 13 Validation loss: 0.046925 Best loss: 0.035479 Accuracy: 98.98% 14 Validation loss: 0.051271 Best loss: 0.035479 Accuracy: 98.75% 15 Validation loss: 0.052683 Best loss: 0.035479 Accuracy: 98.75% 16 Validation loss: 0.044916 Best loss: 0.035479 Accuracy: 98.94% 17 Validation loss: 0.040556 Best loss: 0.035479 Accuracy: 98.87% 18 Validation loss: 0.047525 Best loss: 0.035479 Accuracy: 98.79% 19 Validation loss: 0.055830 Best loss: 0.035479 Accuracy: 98.98% 20 Validation loss: 0.056420 Best loss: 0.035479 Accuracy: 98.32% 21 Validation loss: 0.048499 Best loss: 0.035479 Accuracy: 98.91% 22 Validation loss: 0.056787 Best loss: 0.035479 Accuracy: 98.67% 23 Validation loss: 0.064257 Best loss: 0.035479 Accuracy: 98.79% 24 Validation loss: 0.048222 Best loss: 0.035479 Accuracy: 98.87% 25 Validation loss: 0.046214 Best loss: 0.035479 Accuracy: 98.94% 26 Validation loss: 0.048283 Best loss: 0.035479 Accuracy: 99.02% 27 Validation loss: 0.051521 Best loss: 0.035479 Accuracy: 98.98% 28 Validation loss: 0.047411 Best loss: 0.035479 Accuracy: 98.79% 29 Validation loss: 0.047771 Best loss: 0.035479 Accuracy: 98.91% 30 Validation loss: 0.049030 Best loss: 0.035479 Accuracy: 98.91% 31 Validation loss: 0.052925 Best loss: 0.035479 Accuracy: 99.02% 32 Validation loss: 0.052155 Best loss: 0.035479 Accuracy: 99.06% 33 Validation loss: 0.051572 Best loss: 0.035479 Accuracy: 98.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01, total= 12.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01 0 Validation loss: 0.056979 Best loss: 0.056979 Accuracy: 98.08% 1 Validation loss: 0.045357 Best loss: 0.045357 Accuracy: 98.67% 2 Validation loss: 0.048206 Best loss: 0.045357 Accuracy: 98.55% 3 Validation loss: 0.040019 Best loss: 0.040019 Accuracy: 98.48% 4 Validation loss: 0.036850 Best loss: 0.036850 Accuracy: 98.75% 5 Validation loss: 0.052556 Best loss: 0.036850 Accuracy: 98.40% 6 Validation loss: 0.048808 Best loss: 0.036850 Accuracy: 98.63% 7 Validation loss: 0.044552 Best loss: 0.036850 Accuracy: 98.83% 8 Validation loss: 0.044026 Best loss: 0.036850 Accuracy: 98.67% 9 Validation loss: 0.038270 Best loss: 0.036850 Accuracy: 99.02% 10 Validation loss: 0.045960 Best loss: 0.036850 Accuracy: 98.67% 11 Validation loss: 0.032769 Best loss: 0.032769 Accuracy: 99.02% 12 Validation loss: 0.041136 Best loss: 0.032769 Accuracy: 98.75% 13 Validation loss: 0.035949 Best loss: 0.032769 Accuracy: 99.14% 14 Validation loss: 0.037809 Best loss: 0.032769 Accuracy: 98.87% 15 Validation loss: 0.033936 Best loss: 0.032769 Accuracy: 99.10% 16 Validation loss: 0.051878 Best loss: 0.032769 Accuracy: 98.75% 17 Validation loss: 0.038089 Best loss: 0.032769 Accuracy: 99.14% 18 Validation loss: 0.033791 Best loss: 0.032769 Accuracy: 99.34% 19 Validation loss: 0.030139 Best loss: 0.030139 Accuracy: 99.34% 20 Validation loss: 0.042725 Best loss: 0.030139 Accuracy: 98.94% 21 Validation loss: 0.052613 Best loss: 0.030139 Accuracy: 98.63% 22 Validation loss: 0.040886 Best loss: 0.030139 Accuracy: 98.83% 23 Validation loss: 0.052385 Best loss: 0.030139 Accuracy: 98.98% 24 Validation loss: 0.034357 Best loss: 0.030139 Accuracy: 99.14% 25 Validation loss: 0.053469 Best loss: 0.030139 Accuracy: 98.75% 26 Validation loss: 0.051148 Best loss: 0.030139 Accuracy: 98.79% 27 Validation loss: 0.051416 Best loss: 0.030139 Accuracy: 98.83% 28 Validation loss: 0.045923 Best loss: 0.030139 Accuracy: 98.83% 29 Validation loss: 0.034363 Best loss: 0.030139 Accuracy: 99.14% 30 Validation loss: 0.045930 Best loss: 0.030139 Accuracy: 98.87% 31 Validation loss: 0.040485 Best loss: 0.030139 Accuracy: 98.91% 32 Validation loss: 0.045838 Best loss: 0.030139 Accuracy: 99.06% 33 Validation loss: 0.049657 Best loss: 0.030139 Accuracy: 98.98% 34 Validation loss: 0.043755 Best loss: 0.030139 Accuracy: 99.14% 35 Validation loss: 0.057676 Best loss: 0.030139 Accuracy: 98.79% 36 Validation loss: 0.041608 Best loss: 0.030139 Accuracy: 99.10% 37 Validation loss: 0.046384 Best loss: 0.030139 Accuracy: 99.10% 38 Validation loss: 0.051075 Best loss: 0.030139 Accuracy: 99.02% 39 Validation loss: 0.050342 Best loss: 0.030139 Accuracy: 99.14% 40 Validation loss: 0.038895 Best loss: 0.030139 Accuracy: 99.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=140, batch_size=500, batch_norm_momentum=0.9, learning_rate=0.01, total= 15.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1 0 Validation loss: 1.742093 Best loss: 1.742093 Accuracy: 91.28% 1 Validation loss: 1.055766 Best loss: 1.055766 Accuracy: 88.62% 2 Validation loss: 2.312727 Best loss: 1.055766 Accuracy: 90.23% 3 Validation loss: 0.987924 Best loss: 0.987924 Accuracy: 92.18% 4 Validation loss: 0.190066 Best loss: 0.190066 Accuracy: 96.79% 5 Validation loss: 0.959484 Best loss: 0.190066 Accuracy: 88.04% 6 Validation loss: 0.312340 Best loss: 0.190066 Accuracy: 97.07% 7 Validation loss: 0.242851 Best loss: 0.190066 Accuracy: 95.43% 8 Validation loss: 0.314214 Best loss: 0.190066 Accuracy: 97.65% 9 Validation loss: 0.417402 Best loss: 0.190066 Accuracy: 95.23% 10 Validation loss: 0.178976 Best loss: 0.178976 Accuracy: 97.62% 11 Validation loss: 0.221732 Best loss: 0.178976 Accuracy: 97.73% 12 Validation loss: 0.311403 Best loss: 0.178976 Accuracy: 96.44% 13 Validation loss: 0.493341 Best loss: 0.178976 Accuracy: 96.25% 14 Validation loss: 0.530673 Best loss: 0.178976 Accuracy: 96.17% 15 Validation loss: 0.779975 Best loss: 0.178976 Accuracy: 95.23% 16 Validation loss: 0.282583 Best loss: 0.178976 Accuracy: 97.26% 17 Validation loss: 0.125992 Best loss: 0.125992 Accuracy: 98.36% 18 Validation loss: 0.248297 Best loss: 0.125992 Accuracy: 97.97% 19 Validation loss: 2.712080 Best loss: 0.125992 Accuracy: 76.94% 20 Validation loss: 0.209392 Best loss: 0.125992 Accuracy: 98.01% 21 Validation loss: 0.394147 Best loss: 0.125992 Accuracy: 96.09% 22 Validation loss: 0.510338 Best loss: 0.125992 Accuracy: 97.77% 23 Validation loss: 0.280103 Best loss: 0.125992 Accuracy: 97.73% 24 Validation loss: 0.192012 Best loss: 0.125992 Accuracy: 98.59% 25 Validation loss: 0.274682 Best loss: 0.125992 Accuracy: 98.32% 26 Validation loss: 0.226346 Best loss: 0.125992 Accuracy: 98.24% 27 Validation loss: 0.496828 Best loss: 0.125992 Accuracy: 97.50% 28 Validation loss: 0.246072 Best loss: 0.125992 Accuracy: 97.97% 29 Validation loss: 1.138451 Best loss: 0.125992 Accuracy: 95.04% 30 Validation loss: 0.317270 Best loss: 0.125992 Accuracy: 97.58% 31 Validation loss: 0.399023 Best loss: 0.125992 Accuracy: 97.97% 32 Validation loss: 0.349367 Best loss: 0.125992 Accuracy: 98.20% 33 Validation loss: 0.338811 Best loss: 0.125992 Accuracy: 98.08% 34 Validation loss: 0.231812 Best loss: 0.125992 Accuracy: 98.63% 35 Validation loss: 0.473600 Best loss: 0.125992 Accuracy: 97.89% 36 Validation loss: 0.258060 Best loss: 0.125992 Accuracy: 98.75% 37 Validation loss: 0.292105 Best loss: 0.125992 Accuracy: 98.79% 38 Validation loss: 0.275215 Best loss: 0.125992 Accuracy: 98.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1, total= 7.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1 0 Validation loss: 1.368465 Best loss: 1.368465 Accuracy: 93.78% 1 Validation loss: 12.809568 Best loss: 1.368465 Accuracy: 83.31% 2 Validation loss: 0.432141 Best loss: 0.432141 Accuracy: 96.33% 3 Validation loss: 0.932542 Best loss: 0.432141 Accuracy: 93.55% 4 Validation loss: 0.849091 Best loss: 0.432141 Accuracy: 90.73% 5 Validation loss: 0.711127 Best loss: 0.432141 Accuracy: 94.64% 6 Validation loss: 0.252254 Best loss: 0.252254 Accuracy: 96.72% 7 Validation loss: 0.643157 Best loss: 0.252254 Accuracy: 95.39% 8 Validation loss: 0.212467 Best loss: 0.212467 Accuracy: 98.01% 9 Validation loss: 0.351864 Best loss: 0.212467 Accuracy: 97.26% 10 Validation loss: 0.315193 Best loss: 0.212467 Accuracy: 97.22% 11 Validation loss: 0.304491 Best loss: 0.212467 Accuracy: 96.13% 12 Validation loss: 4.100076 Best loss: 0.212467 Accuracy: 89.37% 13 Validation loss: 0.329181 Best loss: 0.212467 Accuracy: 97.30% 14 Validation loss: 0.169952 Best loss: 0.169952 Accuracy: 97.77% 15 Validation loss: 0.416117 Best loss: 0.169952 Accuracy: 96.87% 16 Validation loss: 0.258601 Best loss: 0.169952 Accuracy: 97.65% 17 Validation loss: 0.257401 Best loss: 0.169952 Accuracy: 98.01% 18 Validation loss: 0.242927 Best loss: 0.169952 Accuracy: 98.44% 19 Validation loss: 0.226116 Best loss: 0.169952 Accuracy: 97.73% 20 Validation loss: 0.210789 Best loss: 0.169952 Accuracy: 98.08% 21 Validation loss: 0.217406 Best loss: 0.169952 Accuracy: 98.01% 22 Validation loss: 0.246331 Best loss: 0.169952 Accuracy: 97.07% 23 Validation loss: 0.859666 Best loss: 0.169952 Accuracy: 93.28% 24 Validation loss: 0.178701 Best loss: 0.169952 Accuracy: 97.77% 25 Validation loss: 0.271729 Best loss: 0.169952 Accuracy: 98.32% 26 Validation loss: 0.262890 Best loss: 0.169952 Accuracy: 98.32% 27 Validation loss: 0.233386 Best loss: 0.169952 Accuracy: 97.34% 28 Validation loss: 0.258193 Best loss: 0.169952 Accuracy: 98.24% 29 Validation loss: 0.263289 Best loss: 0.169952 Accuracy: 98.01% 30 Validation loss: 0.271906 Best loss: 0.169952 Accuracy: 97.81% 31 Validation loss: 0.329723 Best loss: 0.169952 Accuracy: 98.40% 32 Validation loss: 0.262468 Best loss: 0.169952 Accuracy: 98.63% 33 Validation loss: 0.408538 Best loss: 0.169952 Accuracy: 97.50% 34 Validation loss: 0.321472 Best loss: 0.169952 Accuracy: 98.40% 35 Validation loss: 0.278295 Best loss: 0.169952 Accuracy: 98.71% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1, total= 6.9min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1 0 Validation loss: 5.574201 Best loss: 5.574201 Accuracy: 81.70% 1 Validation loss: 1.245204 Best loss: 1.245204 Accuracy: 95.78% 2 Validation loss: 0.444110 Best loss: 0.444110 Accuracy: 96.52% 3 Validation loss: 1.443920 Best loss: 0.444110 Accuracy: 90.03% 4 Validation loss: 1.315827 Best loss: 0.444110 Accuracy: 93.12% 5 Validation loss: 0.649003 Best loss: 0.444110 Accuracy: 91.05% 6 Validation loss: 0.395552 Best loss: 0.395552 Accuracy: 94.92% 7 Validation loss: 0.342828 Best loss: 0.342828 Accuracy: 95.35% 8 Validation loss: 0.138084 Best loss: 0.138084 Accuracy: 97.93% 9 Validation loss: 0.482633 Best loss: 0.138084 Accuracy: 94.06% 10 Validation loss: 0.255667 Best loss: 0.138084 Accuracy: 96.52% 11 Validation loss: 0.877179 Best loss: 0.138084 Accuracy: 95.27% 12 Validation loss: 0.158146 Best loss: 0.138084 Accuracy: 98.08% 13 Validation loss: 0.256444 Best loss: 0.138084 Accuracy: 96.33% 14 Validation loss: 0.141785 Best loss: 0.138084 Accuracy: 98.44% 15 Validation loss: 0.303284 Best loss: 0.138084 Accuracy: 97.65% 16 Validation loss: 0.211630 Best loss: 0.138084 Accuracy: 97.62% 17 Validation loss: 0.488440 Best loss: 0.138084 Accuracy: 97.11% 18 Validation loss: 0.101030 Best loss: 0.101030 Accuracy: 98.91% 19 Validation loss: 0.242470 Best loss: 0.101030 Accuracy: 98.28% 20 Validation loss: 0.233681 Best loss: 0.101030 Accuracy: 97.38% 21 Validation loss: 0.163727 Best loss: 0.101030 Accuracy: 97.81% 22 Validation loss: 0.151497 Best loss: 0.101030 Accuracy: 98.44% 23 Validation loss: 0.283031 Best loss: 0.101030 Accuracy: 96.99% 24 Validation loss: 0.191807 Best loss: 0.101030 Accuracy: 98.40% 25 Validation loss: 0.392310 Best loss: 0.101030 Accuracy: 97.11% 26 Validation loss: 0.264641 Best loss: 0.101030 Accuracy: 98.44% 27 Validation loss: 0.427995 Best loss: 0.101030 Accuracy: 96.13% 28 Validation loss: 0.177024 Best loss: 0.101030 Accuracy: 98.71% 29 Validation loss: 0.217375 Best loss: 0.101030 Accuracy: 97.85% 30 Validation loss: 1.188633 Best loss: 0.101030 Accuracy: 94.41% 31 Validation loss: 0.202269 Best loss: 0.101030 Accuracy: 98.71% 32 Validation loss: 0.479391 Best loss: 0.101030 Accuracy: 96.91% 33 Validation loss: 0.390757 Best loss: 0.101030 Accuracy: 97.77% 34 Validation loss: 0.455240 Best loss: 0.101030 Accuracy: 97.89% 35 Validation loss: 0.287360 Best loss: 0.101030 Accuracy: 97.85% 36 Validation loss: 0.333966 Best loss: 0.101030 Accuracy: 97.15% 37 Validation loss: 0.310145 Best loss: 0.101030 Accuracy: 98.48% 38 Validation loss: 0.269206 Best loss: 0.101030 Accuracy: 98.32% 39 Validation loss: 0.357182 Best loss: 0.101030 Accuracy: 98.24% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=160, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.1, total= 7.5min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.133884 Best loss: 0.133884 Accuracy: 97.03% 1 Validation loss: 0.115825 Best loss: 0.115825 Accuracy: 97.93% 2 Validation loss: 0.118352 Best loss: 0.115825 Accuracy: 97.03% 3 Validation loss: 0.053005 Best loss: 0.053005 Accuracy: 98.40% 4 Validation loss: 0.049954 Best loss: 0.049954 Accuracy: 98.75% 5 Validation loss: 0.056778 Best loss: 0.049954 Accuracy: 98.51% 6 Validation loss: 0.069246 Best loss: 0.049954 Accuracy: 98.12% 7 Validation loss: 0.075179 Best loss: 0.049954 Accuracy: 97.93% 8 Validation loss: 0.070619 Best loss: 0.049954 Accuracy: 98.16% 9 Validation loss: 0.050182 Best loss: 0.049954 Accuracy: 98.63% 10 Validation loss: 0.053977 Best loss: 0.049954 Accuracy: 98.75% 11 Validation loss: 0.049798 Best loss: 0.049798 Accuracy: 98.91% 12 Validation loss: 0.046423 Best loss: 0.046423 Accuracy: 99.02% 13 Validation loss: 0.048498 Best loss: 0.046423 Accuracy: 98.83% 14 Validation loss: 0.053631 Best loss: 0.046423 Accuracy: 98.94% 15 Validation loss: 0.045984 Best loss: 0.045984 Accuracy: 98.98% 16 Validation loss: 0.069705 Best loss: 0.045984 Accuracy: 98.51% 17 Validation loss: 0.048883 Best loss: 0.045984 Accuracy: 98.79% 18 Validation loss: 0.046947 Best loss: 0.045984 Accuracy: 98.79% 19 Validation loss: 0.068785 Best loss: 0.045984 Accuracy: 98.71% 20 Validation loss: 0.078370 Best loss: 0.045984 Accuracy: 98.48% 21 Validation loss: 0.057828 Best loss: 0.045984 Accuracy: 98.67% 22 Validation loss: 0.049785 Best loss: 0.045984 Accuracy: 99.02% 23 Validation loss: 0.066968 Best loss: 0.045984 Accuracy: 98.55% 24 Validation loss: 0.062439 Best loss: 0.045984 Accuracy: 98.63% 25 Validation loss: 0.055275 Best loss: 0.045984 Accuracy: 98.83% 26 Validation loss: 0.071204 Best loss: 0.045984 Accuracy: 98.55% 27 Validation loss: 0.064278 Best loss: 0.045984 Accuracy: 98.59% 28 Validation loss: 0.063539 Best loss: 0.045984 Accuracy: 98.98% 29 Validation loss: 0.048904 Best loss: 0.045984 Accuracy: 99.10% 30 Validation loss: 0.075003 Best loss: 0.045984 Accuracy: 98.20% 31 Validation loss: 0.078903 Best loss: 0.045984 Accuracy: 98.59% 32 Validation loss: 0.040982 Best loss: 0.040982 Accuracy: 98.94% 33 Validation loss: 0.042294 Best loss: 0.040982 Accuracy: 99.10% 34 Validation loss: 0.051099 Best loss: 0.040982 Accuracy: 99.02% 35 Validation loss: 0.033899 Best loss: 0.033899 Accuracy: 99.26% 36 Validation loss: 0.030725 Best loss: 0.030725 Accuracy: 99.30% 37 Validation loss: 0.039919 Best loss: 0.030725 Accuracy: 99.26% 38 Validation loss: 0.048509 Best loss: 0.030725 Accuracy: 98.94% 39 Validation loss: 0.060102 Best loss: 0.030725 Accuracy: 98.67% 40 Validation loss: 0.039253 Best loss: 0.030725 Accuracy: 98.98% 41 Validation loss: 0.030763 Best loss: 0.030725 Accuracy: 99.30% 42 Validation loss: 0.039731 Best loss: 0.030725 Accuracy: 99.30% 43 Validation loss: 0.068669 Best loss: 0.030725 Accuracy: 98.59% 44 Validation loss: 0.047460 Best loss: 0.030725 Accuracy: 99.02% 45 Validation loss: 0.046375 Best loss: 0.030725 Accuracy: 99.22% 46 Validation loss: 0.061973 Best loss: 0.030725 Accuracy: 98.83% 47 Validation loss: 0.072355 Best loss: 0.030725 Accuracy: 98.16% 48 Validation loss: 0.039004 Best loss: 0.030725 Accuracy: 99.22% 49 Validation loss: 0.037060 Best loss: 0.030725 Accuracy: 99.30% 50 Validation loss: 0.046441 Best loss: 0.030725 Accuracy: 99.14% 51 Validation loss: 0.050854 Best loss: 0.030725 Accuracy: 98.98% 52 Validation loss: 0.123935 Best loss: 0.030725 Accuracy: 98.24% 53 Validation loss: 0.056133 Best loss: 0.030725 Accuracy: 98.63% 54 Validation loss: 0.071320 Best loss: 0.030725 Accuracy: 98.59% 55 Validation loss: 0.057849 Best loss: 0.030725 Accuracy: 98.94% 56 Validation loss: 0.068682 Best loss: 0.030725 Accuracy: 98.79% 57 Validation loss: 0.040209 Best loss: 0.030725 Accuracy: 99.18% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 1.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.196061 Best loss: 0.196061 Accuracy: 96.17% 1 Validation loss: 0.063771 Best loss: 0.063771 Accuracy: 98.32% 2 Validation loss: 0.063669 Best loss: 0.063669 Accuracy: 98.36% 3 Validation loss: 0.073477 Best loss: 0.063669 Accuracy: 98.32% 4 Validation loss: 0.069188 Best loss: 0.063669 Accuracy: 98.20% 5 Validation loss: 0.036731 Best loss: 0.036731 Accuracy: 98.83% 6 Validation loss: 0.054120 Best loss: 0.036731 Accuracy: 98.48% 7 Validation loss: 0.060699 Best loss: 0.036731 Accuracy: 98.55% 8 Validation loss: 0.050817 Best loss: 0.036731 Accuracy: 98.94% 9 Validation loss: 0.053192 Best loss: 0.036731 Accuracy: 98.71% 10 Validation loss: 0.061237 Best loss: 0.036731 Accuracy: 98.32% 11 Validation loss: 0.054445 Best loss: 0.036731 Accuracy: 98.59% 12 Validation loss: 0.045724 Best loss: 0.036731 Accuracy: 98.87% 13 Validation loss: 0.065591 Best loss: 0.036731 Accuracy: 98.55% 14 Validation loss: 0.056691 Best loss: 0.036731 Accuracy: 98.91% 15 Validation loss: 0.039316 Best loss: 0.036731 Accuracy: 99.22% 16 Validation loss: 0.060962 Best loss: 0.036731 Accuracy: 98.67% 17 Validation loss: 0.045373 Best loss: 0.036731 Accuracy: 98.91% 18 Validation loss: 0.064475 Best loss: 0.036731 Accuracy: 98.71% 19 Validation loss: 0.042589 Best loss: 0.036731 Accuracy: 99.10% 20 Validation loss: 0.048400 Best loss: 0.036731 Accuracy: 98.98% 21 Validation loss: 0.060045 Best loss: 0.036731 Accuracy: 99.06% 22 Validation loss: 0.048115 Best loss: 0.036731 Accuracy: 99.10% 23 Validation loss: 0.110987 Best loss: 0.036731 Accuracy: 98.24% 24 Validation loss: 0.051041 Best loss: 0.036731 Accuracy: 99.06% 25 Validation loss: 0.064848 Best loss: 0.036731 Accuracy: 98.94% 26 Validation loss: 0.066829 Best loss: 0.036731 Accuracy: 98.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 37.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01 0 Validation loss: 0.145364 Best loss: 0.145364 Accuracy: 96.99% 1 Validation loss: 0.062563 Best loss: 0.062563 Accuracy: 98.28% 2 Validation loss: 0.063061 Best loss: 0.062563 Accuracy: 97.85% 3 Validation loss: 0.059503 Best loss: 0.059503 Accuracy: 98.59% 4 Validation loss: 0.055112 Best loss: 0.055112 Accuracy: 98.67% 5 Validation loss: 0.060087 Best loss: 0.055112 Accuracy: 98.40% 6 Validation loss: 0.045368 Best loss: 0.045368 Accuracy: 98.51% 7 Validation loss: 0.090626 Best loss: 0.045368 Accuracy: 98.20% 8 Validation loss: 0.062201 Best loss: 0.045368 Accuracy: 98.67% 9 Validation loss: 0.056740 Best loss: 0.045368 Accuracy: 98.87% 10 Validation loss: 0.038689 Best loss: 0.038689 Accuracy: 98.87% 11 Validation loss: 0.043084 Best loss: 0.038689 Accuracy: 98.98% 12 Validation loss: 0.037991 Best loss: 0.037991 Accuracy: 99.10% 13 Validation loss: 0.044937 Best loss: 0.037991 Accuracy: 98.63% 14 Validation loss: 0.054282 Best loss: 0.037991 Accuracy: 98.63% 15 Validation loss: 0.057419 Best loss: 0.037991 Accuracy: 98.83% 16 Validation loss: 0.050489 Best loss: 0.037991 Accuracy: 98.87% 17 Validation loss: 0.048947 Best loss: 0.037991 Accuracy: 98.91% 18 Validation loss: 0.052636 Best loss: 0.037991 Accuracy: 99.02% 19 Validation loss: 0.058444 Best loss: 0.037991 Accuracy: 98.79% 20 Validation loss: 0.061106 Best loss: 0.037991 Accuracy: 98.94% 21 Validation loss: 0.042630 Best loss: 0.037991 Accuracy: 99.18% 22 Validation loss: 0.040157 Best loss: 0.037991 Accuracy: 98.94% 23 Validation loss: 0.054595 Best loss: 0.037991 Accuracy: 98.87% 24 Validation loss: 0.053633 Best loss: 0.037991 Accuracy: 98.59% 25 Validation loss: 0.042523 Best loss: 0.037991 Accuracy: 98.79% 26 Validation loss: 0.046764 Best loss: 0.037991 Accuracy: 98.87% 27 Validation loss: 0.060872 Best loss: 0.037991 Accuracy: 98.91% 28 Validation loss: 0.042604 Best loss: 0.037991 Accuracy: 99.14% 29 Validation loss: 0.056780 Best loss: 0.037991 Accuracy: 98.94% 30 Validation loss: 0.061686 Best loss: 0.037991 Accuracy: 99.06% 31 Validation loss: 0.040676 Best loss: 0.037991 Accuracy: 99.10% 32 Validation loss: 0.044714 Best loss: 0.037991 Accuracy: 99.06% 33 Validation loss: 0.044060 Best loss: 0.037991 Accuracy: 99.14% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.01, total= 46.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.185942 Best loss: 0.185942 Accuracy: 93.86% 1 Validation loss: 0.153762 Best loss: 0.153762 Accuracy: 94.45% 2 Validation loss: 0.123385 Best loss: 0.123385 Accuracy: 96.95% 3 Validation loss: 0.148041 Best loss: 0.123385 Accuracy: 94.68% 4 Validation loss: 0.086911 Best loss: 0.086911 Accuracy: 97.73% 5 Validation loss: 0.077310 Best loss: 0.077310 Accuracy: 98.01% 6 Validation loss: 0.103754 Best loss: 0.077310 Accuracy: 97.26% 7 Validation loss: 0.074335 Best loss: 0.074335 Accuracy: 97.77% 8 Validation loss: 0.079664 Best loss: 0.074335 Accuracy: 97.73% 9 Validation loss: 0.106884 Best loss: 0.074335 Accuracy: 97.54% 10 Validation loss: 0.071167 Best loss: 0.071167 Accuracy: 98.36% 11 Validation loss: 0.062629 Best loss: 0.062629 Accuracy: 98.59% 12 Validation loss: 0.069027 Best loss: 0.062629 Accuracy: 98.40% 13 Validation loss: 0.074329 Best loss: 0.062629 Accuracy: 97.58% 14 Validation loss: 0.059478 Best loss: 0.059478 Accuracy: 98.51% 15 Validation loss: 0.094557 Best loss: 0.059478 Accuracy: 98.51% 16 Validation loss: 0.095280 Best loss: 0.059478 Accuracy: 98.24% 17 Validation loss: 0.096403 Best loss: 0.059478 Accuracy: 98.20% 18 Validation loss: 0.081009 Best loss: 0.059478 Accuracy: 98.16% 19 Validation loss: 0.090945 Best loss: 0.059478 Accuracy: 98.12% 20 Validation loss: 0.089775 Best loss: 0.059478 Accuracy: 97.89% 21 Validation loss: 0.080708 Best loss: 0.059478 Accuracy: 97.97% 22 Validation loss: 0.089445 Best loss: 0.059478 Accuracy: 98.20% 23 Validation loss: 0.073173 Best loss: 0.059478 Accuracy: 98.75% 24 Validation loss: 0.098331 Best loss: 0.059478 Accuracy: 97.03% 25 Validation loss: 0.056277 Best loss: 0.056277 Accuracy: 98.63% 26 Validation loss: 0.062644 Best loss: 0.056277 Accuracy: 98.36% 27 Validation loss: 0.086439 Best loss: 0.056277 Accuracy: 98.12% 28 Validation loss: 0.063115 Best loss: 0.056277 Accuracy: 98.55% 29 Validation loss: 0.091627 Best loss: 0.056277 Accuracy: 98.28% 30 Validation loss: 0.076686 Best loss: 0.056277 Accuracy: 98.32% 31 Validation loss: 0.094946 Best loss: 0.056277 Accuracy: 97.93% 32 Validation loss: 0.069201 Best loss: 0.056277 Accuracy: 98.83% 33 Validation loss: 0.061430 Best loss: 0.056277 Accuracy: 98.24% 34 Validation loss: 0.083254 Best loss: 0.056277 Accuracy: 98.51% 35 Validation loss: 0.146353 Best loss: 0.056277 Accuracy: 98.05% 36 Validation loss: 0.062524 Best loss: 0.056277 Accuracy: 98.87% 37 Validation loss: 0.067914 Best loss: 0.056277 Accuracy: 98.63% 38 Validation loss: 0.070633 Best loss: 0.056277 Accuracy: 98.36% 39 Validation loss: 0.073477 Best loss: 0.056277 Accuracy: 98.51% 40 Validation loss: 0.077894 Best loss: 0.056277 Accuracy: 98.51% 41 Validation loss: 0.063132 Best loss: 0.056277 Accuracy: 98.32% 42 Validation loss: 0.080248 Best loss: 0.056277 Accuracy: 98.55% 43 Validation loss: 0.085439 Best loss: 0.056277 Accuracy: 98.05% 44 Validation loss: 0.087227 Best loss: 0.056277 Accuracy: 98.44% 45 Validation loss: 0.169515 Best loss: 0.056277 Accuracy: 96.13% 46 Validation loss: 0.222734 Best loss: 0.056277 Accuracy: 96.68% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1, total= 8.8min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.148731 Best loss: 0.148731 Accuracy: 95.97% 1 Validation loss: 0.145826 Best loss: 0.145826 Accuracy: 96.21% 2 Validation loss: 0.098106 Best loss: 0.098106 Accuracy: 97.11% 3 Validation loss: 0.308717 Best loss: 0.098106 Accuracy: 95.00% 4 Validation loss: 0.065391 Best loss: 0.065391 Accuracy: 98.32% 5 Validation loss: 0.098380 Best loss: 0.065391 Accuracy: 97.38% 6 Validation loss: 0.059383 Best loss: 0.059383 Accuracy: 98.36% 7 Validation loss: 0.080345 Best loss: 0.059383 Accuracy: 97.81% 8 Validation loss: 0.146474 Best loss: 0.059383 Accuracy: 96.52% 9 Validation loss: 0.107901 Best loss: 0.059383 Accuracy: 97.34% 10 Validation loss: 0.067520 Best loss: 0.059383 Accuracy: 98.12% 11 Validation loss: 0.127893 Best loss: 0.059383 Accuracy: 97.89% 12 Validation loss: 0.115032 Best loss: 0.059383 Accuracy: 97.81% 13 Validation loss: 0.080025 Best loss: 0.059383 Accuracy: 98.20% 14 Validation loss: 0.065810 Best loss: 0.059383 Accuracy: 98.36% 15 Validation loss: 0.122632 Best loss: 0.059383 Accuracy: 96.33% 16 Validation loss: 0.177292 Best loss: 0.059383 Accuracy: 95.35% 17 Validation loss: 0.107256 Best loss: 0.059383 Accuracy: 97.93% 18 Validation loss: 0.072756 Best loss: 0.059383 Accuracy: 98.44% 19 Validation loss: 0.063022 Best loss: 0.059383 Accuracy: 98.71% 20 Validation loss: 0.084522 Best loss: 0.059383 Accuracy: 98.05% 21 Validation loss: 0.070288 Best loss: 0.059383 Accuracy: 98.36% 22 Validation loss: 0.070839 Best loss: 0.059383 Accuracy: 98.36% 23 Validation loss: 0.049607 Best loss: 0.049607 Accuracy: 98.71% 24 Validation loss: 0.126961 Best loss: 0.049607 Accuracy: 97.38% 25 Validation loss: 0.108683 Best loss: 0.049607 Accuracy: 98.44% 26 Validation loss: 0.111404 Best loss: 0.049607 Accuracy: 97.07% 27 Validation loss: 0.066326 Best loss: 0.049607 Accuracy: 98.40% 28 Validation loss: 0.082034 Best loss: 0.049607 Accuracy: 98.36% 29 Validation loss: 0.065672 Best loss: 0.049607 Accuracy: 98.91% 30 Validation loss: 0.070105 Best loss: 0.049607 Accuracy: 98.28% 31 Validation loss: 0.075912 Best loss: 0.049607 Accuracy: 97.85% 32 Validation loss: 0.079841 Best loss: 0.049607 Accuracy: 98.51% 33 Validation loss: 0.067589 Best loss: 0.049607 Accuracy: 98.44% 34 Validation loss: 0.071396 Best loss: 0.049607 Accuracy: 98.59% 35 Validation loss: 0.061299 Best loss: 0.049607 Accuracy: 98.79% 36 Validation loss: 0.055520 Best loss: 0.049607 Accuracy: 98.75% 37 Validation loss: 0.111776 Best loss: 0.049607 Accuracy: 97.62% 38 Validation loss: 0.050153 Best loss: 0.049607 Accuracy: 98.71% 39 Validation loss: 0.090981 Best loss: 0.049607 Accuracy: 98.63% 40 Validation loss: 0.046593 Best loss: 0.046593 Accuracy: 99.02% 41 Validation loss: 0.281856 Best loss: 0.046593 Accuracy: 96.83% 42 Validation loss: 0.044223 Best loss: 0.044223 Accuracy: 98.71% 43 Validation loss: 0.117946 Best loss: 0.044223 Accuracy: 97.97% 44 Validation loss: 0.047351 Best loss: 0.044223 Accuracy: 98.79% 45 Validation loss: 0.115456 Best loss: 0.044223 Accuracy: 97.58% 46 Validation loss: 0.128656 Best loss: 0.044223 Accuracy: 98.20% 47 Validation loss: 0.104774 Best loss: 0.044223 Accuracy: 98.01% 48 Validation loss: 0.093181 Best loss: 0.044223 Accuracy: 98.51% 49 Validation loss: 0.102467 Best loss: 0.044223 Accuracy: 98.40% 50 Validation loss: 0.071877 Best loss: 0.044223 Accuracy: 98.79% 51 Validation loss: 0.085424 Best loss: 0.044223 Accuracy: 98.44% 52 Validation loss: 0.079334 Best loss: 0.044223 Accuracy: 98.59% 53 Validation loss: 0.081534 Best loss: 0.044223 Accuracy: 98.87% 54 Validation loss: 0.134484 Best loss: 0.044223 Accuracy: 97.81% 55 Validation loss: 0.113530 Best loss: 0.044223 Accuracy: 98.44% 56 Validation loss: 0.068482 Best loss: 0.044223 Accuracy: 98.83% 57 Validation loss: 0.087978 Best loss: 0.044223 Accuracy: 98.79% 58 Validation loss: 0.068166 Best loss: 0.044223 Accuracy: 98.83% 59 Validation loss: 0.071825 Best loss: 0.044223 Accuracy: 98.20% 60 Validation loss: 0.123198 Best loss: 0.044223 Accuracy: 98.05% 61 Validation loss: 0.095200 Best loss: 0.044223 Accuracy: 98.40% 62 Validation loss: 0.067082 Best loss: 0.044223 Accuracy: 98.63% 63 Validation loss: 0.057428 Best loss: 0.044223 Accuracy: 98.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1, total=11.9min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1 0 Validation loss: 0.125935 Best loss: 0.125935 Accuracy: 96.17% 1 Validation loss: 0.106976 Best loss: 0.106976 Accuracy: 96.83% 2 Validation loss: 0.096016 Best loss: 0.096016 Accuracy: 97.11% 3 Validation loss: 0.087471 Best loss: 0.087471 Accuracy: 96.72% 4 Validation loss: 0.117183 Best loss: 0.087471 Accuracy: 96.40% 5 Validation loss: 0.111497 Best loss: 0.087471 Accuracy: 97.11% 6 Validation loss: 0.090927 Best loss: 0.087471 Accuracy: 97.50% 7 Validation loss: 0.076401 Best loss: 0.076401 Accuracy: 97.65% 8 Validation loss: 0.062935 Best loss: 0.062935 Accuracy: 98.36% 9 Validation loss: 0.063184 Best loss: 0.062935 Accuracy: 98.28% 10 Validation loss: 0.072605 Best loss: 0.062935 Accuracy: 98.20% 11 Validation loss: 0.077420 Best loss: 0.062935 Accuracy: 98.24% 12 Validation loss: 0.079705 Best loss: 0.062935 Accuracy: 97.81% 13 Validation loss: 0.071182 Best loss: 0.062935 Accuracy: 98.24% 14 Validation loss: 0.054861 Best loss: 0.054861 Accuracy: 98.28% 15 Validation loss: 0.080699 Best loss: 0.054861 Accuracy: 98.44% 16 Validation loss: 0.047378 Best loss: 0.047378 Accuracy: 98.59% 17 Validation loss: 0.070404 Best loss: 0.047378 Accuracy: 98.01% 18 Validation loss: 0.052391 Best loss: 0.047378 Accuracy: 98.63% 19 Validation loss: 0.049630 Best loss: 0.047378 Accuracy: 98.79% 20 Validation loss: 0.043048 Best loss: 0.043048 Accuracy: 98.59% 21 Validation loss: 0.139742 Best loss: 0.043048 Accuracy: 97.93% 22 Validation loss: 0.042941 Best loss: 0.042941 Accuracy: 98.75% 23 Validation loss: 0.048975 Best loss: 0.042941 Accuracy: 98.75% 24 Validation loss: 0.052035 Best loss: 0.042941 Accuracy: 98.67% 25 Validation loss: 0.107214 Best loss: 0.042941 Accuracy: 97.69% 26 Validation loss: 0.066760 Best loss: 0.042941 Accuracy: 98.36% 27 Validation loss: 0.129057 Best loss: 0.042941 Accuracy: 96.29% 28 Validation loss: 0.044710 Best loss: 0.042941 Accuracy: 98.75% 29 Validation loss: 0.177816 Best loss: 0.042941 Accuracy: 97.69% 30 Validation loss: 0.094833 Best loss: 0.042941 Accuracy: 98.12% 31 Validation loss: 0.066054 Best loss: 0.042941 Accuracy: 98.40% 32 Validation loss: 0.097292 Best loss: 0.042941 Accuracy: 98.05% 33 Validation loss: 0.052169 Best loss: 0.042941 Accuracy: 98.94% 34 Validation loss: 0.045900 Best loss: 0.042941 Accuracy: 98.63% 35 Validation loss: 0.090128 Best loss: 0.042941 Accuracy: 98.05% 36 Validation loss: 0.119139 Best loss: 0.042941 Accuracy: 97.62% 37 Validation loss: 0.107294 Best loss: 0.042941 Accuracy: 97.73% 38 Validation loss: 0.069148 Best loss: 0.042941 Accuracy: 98.63% 39 Validation loss: 0.063856 Best loss: 0.042941 Accuracy: 98.83% 40 Validation loss: 0.051410 Best loss: 0.042941 Accuracy: 98.83% 41 Validation loss: 0.058026 Best loss: 0.042941 Accuracy: 98.63% 42 Validation loss: 0.158766 Best loss: 0.042941 Accuracy: 98.28% 43 Validation loss: 0.091333 Best loss: 0.042941 Accuracy: 98.32% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=100, batch_size=10, batch_norm_momentum=0.9, learning_rate=0.1, total= 8.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 7.963143 Best loss: 7.963143 Accuracy: 74.04% 1 Validation loss: 0.459478 Best loss: 0.459478 Accuracy: 91.63% 2 Validation loss: 0.304325 Best loss: 0.304325 Accuracy: 95.00% 3 Validation loss: 0.720291 Best loss: 0.304325 Accuracy: 88.82% 4 Validation loss: 0.430557 Best loss: 0.304325 Accuracy: 95.15% 5 Validation loss: 0.437078 Best loss: 0.304325 Accuracy: 89.52% 6 Validation loss: 0.109958 Best loss: 0.109958 Accuracy: 97.34% 7 Validation loss: 0.684968 Best loss: 0.109958 Accuracy: 93.43% 8 Validation loss: 1.872934 Best loss: 0.109958 Accuracy: 85.65% 9 Validation loss: 0.406114 Best loss: 0.109958 Accuracy: 97.15% 10 Validation loss: 0.091563 Best loss: 0.091563 Accuracy: 97.77% 11 Validation loss: 0.103374 Best loss: 0.091563 Accuracy: 97.85% 12 Validation loss: 0.703526 Best loss: 0.091563 Accuracy: 96.21% 13 Validation loss: 0.090695 Best loss: 0.090695 Accuracy: 98.01% 14 Validation loss: 0.458955 Best loss: 0.090695 Accuracy: 95.86% 15 Validation loss: 0.101434 Best loss: 0.090695 Accuracy: 98.44% 16 Validation loss: 0.169162 Best loss: 0.090695 Accuracy: 97.42% 17 Validation loss: 0.120762 Best loss: 0.090695 Accuracy: 97.93% 18 Validation loss: 0.176938 Best loss: 0.090695 Accuracy: 97.73% 19 Validation loss: 0.079539 Best loss: 0.079539 Accuracy: 98.51% 20 Validation loss: 0.077751 Best loss: 0.077751 Accuracy: 98.63% 21 Validation loss: 0.101532 Best loss: 0.077751 Accuracy: 98.24% 22 Validation loss: 0.118215 Best loss: 0.077751 Accuracy: 98.79% 23 Validation loss: 0.133367 Best loss: 0.077751 Accuracy: 98.44% 24 Validation loss: 0.091337 Best loss: 0.077751 Accuracy: 98.75% 25 Validation loss: 0.193105 Best loss: 0.077751 Accuracy: 98.44% 26 Validation loss: 0.184204 Best loss: 0.077751 Accuracy: 97.97% 27 Validation loss: 0.218658 Best loss: 0.077751 Accuracy: 97.42% 28 Validation loss: 0.396900 Best loss: 0.077751 Accuracy: 95.39% 29 Validation loss: 0.209390 Best loss: 0.077751 Accuracy: 98.40% 30 Validation loss: 0.248488 Best loss: 0.077751 Accuracy: 98.16% 31 Validation loss: 0.117062 Best loss: 0.077751 Accuracy: 98.98% 32 Validation loss: 0.348703 Best loss: 0.077751 Accuracy: 97.65% 33 Validation loss: 0.192045 Best loss: 0.077751 Accuracy: 97.85% 34 Validation loss: 0.097144 Best loss: 0.077751 Accuracy: 98.91% 35 Validation loss: 0.145620 Best loss: 0.077751 Accuracy: 98.51% 36 Validation loss: 0.108844 Best loss: 0.077751 Accuracy: 98.98% 37 Validation loss: 0.127959 Best loss: 0.077751 Accuracy: 98.87% 38 Validation loss: 0.188711 Best loss: 0.077751 Accuracy: 98.48% 39 Validation loss: 0.141281 Best loss: 0.077751 Accuracy: 98.87% 40 Validation loss: 0.111381 Best loss: 0.077751 Accuracy: 98.71% 41 Validation loss: 0.453114 Best loss: 0.077751 Accuracy: 96.17% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05, total= 7.0min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 0.631445 Best loss: 0.631445 Accuracy: 94.84% 1 Validation loss: 1.848673 Best loss: 0.631445 Accuracy: 79.01% 2 Validation loss: 0.963000 Best loss: 0.631445 Accuracy: 95.19% 3 Validation loss: 0.232465 Best loss: 0.232465 Accuracy: 97.46% 4 Validation loss: 1.654795 Best loss: 0.232465 Accuracy: 86.20% 5 Validation loss: 1.785381 Best loss: 0.232465 Accuracy: 89.13% 6 Validation loss: 0.183600 Best loss: 0.183600 Accuracy: 95.54% 7 Validation loss: 0.180149 Best loss: 0.180149 Accuracy: 97.65% 8 Validation loss: 0.093468 Best loss: 0.093468 Accuracy: 97.77% 9 Validation loss: 0.128586 Best loss: 0.093468 Accuracy: 98.24% 10 Validation loss: 0.424668 Best loss: 0.093468 Accuracy: 89.25% 11 Validation loss: 0.145463 Best loss: 0.093468 Accuracy: 97.97% 12 Validation loss: 0.102410 Best loss: 0.093468 Accuracy: 98.28% 13 Validation loss: 0.409478 Best loss: 0.093468 Accuracy: 95.27% 14 Validation loss: 0.097525 Best loss: 0.093468 Accuracy: 98.48% 15 Validation loss: 0.072573 Best loss: 0.072573 Accuracy: 98.55% 16 Validation loss: 0.189616 Best loss: 0.072573 Accuracy: 97.85% 17 Validation loss: 0.135486 Best loss: 0.072573 Accuracy: 97.50% 18 Validation loss: 0.258830 Best loss: 0.072573 Accuracy: 96.64% 19 Validation loss: 0.312053 Best loss: 0.072573 Accuracy: 96.60% 20 Validation loss: 0.198905 Best loss: 0.072573 Accuracy: 98.05% 21 Validation loss: 0.097202 Best loss: 0.072573 Accuracy: 98.83% 22 Validation loss: 0.197320 Best loss: 0.072573 Accuracy: 98.05% 23 Validation loss: 0.138977 Best loss: 0.072573 Accuracy: 98.32% 24 Validation loss: 0.126922 Best loss: 0.072573 Accuracy: 98.67% 25 Validation loss: 0.128252 Best loss: 0.072573 Accuracy: 98.87% 26 Validation loss: 0.114236 Best loss: 0.072573 Accuracy: 98.63% 27 Validation loss: 0.130442 Best loss: 0.072573 Accuracy: 98.71% 28 Validation loss: 0.378450 Best loss: 0.072573 Accuracy: 97.46% 29 Validation loss: 0.180170 Best loss: 0.072573 Accuracy: 98.71% 30 Validation loss: 0.321404 Best loss: 0.072573 Accuracy: 98.32% 31 Validation loss: 0.144786 Best loss: 0.072573 Accuracy: 98.55% 32 Validation loss: 0.208575 Best loss: 0.072573 Accuracy: 98.24% 33 Validation loss: 0.178359 Best loss: 0.072573 Accuracy: 98.59% 34 Validation loss: 0.148478 Best loss: 0.072573 Accuracy: 98.79% 35 Validation loss: 0.133787 Best loss: 0.072573 Accuracy: 98.87% 36 Validation loss: 0.326629 Best loss: 0.072573 Accuracy: 98.28% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05, total= 6.3min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05 0 Validation loss: 0.892618 Best loss: 0.892618 Accuracy: 90.77% 1 Validation loss: 0.186619 Best loss: 0.186619 Accuracy: 97.77% 2 Validation loss: 0.160038 Best loss: 0.160038 Accuracy: 97.07% 3 Validation loss: 0.093736 Best loss: 0.093736 Accuracy: 97.77% 4 Validation loss: 0.083296 Best loss: 0.083296 Accuracy: 97.73% 5 Validation loss: 0.127891 Best loss: 0.083296 Accuracy: 98.08% 6 Validation loss: 0.305748 Best loss: 0.083296 Accuracy: 97.85% 7 Validation loss: 0.162286 Best loss: 0.083296 Accuracy: 97.77% 8 Validation loss: 0.055399 Best loss: 0.055399 Accuracy: 98.98% 9 Validation loss: 0.080164 Best loss: 0.055399 Accuracy: 98.08% 10 Validation loss: 0.178106 Best loss: 0.055399 Accuracy: 97.42% 11 Validation loss: 0.202410 Best loss: 0.055399 Accuracy: 97.15% 12 Validation loss: 0.085766 Best loss: 0.055399 Accuracy: 98.28% 13 Validation loss: 0.075104 Best loss: 0.055399 Accuracy: 98.91% 14 Validation loss: 0.211115 Best loss: 0.055399 Accuracy: 98.01% 15 Validation loss: 0.101024 Best loss: 0.055399 Accuracy: 98.12% 16 Validation loss: 0.107289 Best loss: 0.055399 Accuracy: 98.44% 17 Validation loss: 0.131663 Best loss: 0.055399 Accuracy: 98.44% 18 Validation loss: 0.131719 Best loss: 0.055399 Accuracy: 98.16% 19 Validation loss: 0.108998 Best loss: 0.055399 Accuracy: 98.44% 20 Validation loss: 7.860622 Best loss: 0.055399 Accuracy: 78.19% 21 Validation loss: 0.061382 Best loss: 0.055399 Accuracy: 99.26% 22 Validation loss: 0.094452 Best loss: 0.055399 Accuracy: 98.59% 23 Validation loss: 0.161602 Best loss: 0.055399 Accuracy: 97.54% 24 Validation loss: 0.154862 Best loss: 0.055399 Accuracy: 98.08% 25 Validation loss: 0.526251 Best loss: 0.055399 Accuracy: 96.01% 26 Validation loss: 0.095618 Best loss: 0.055399 Accuracy: 98.83% 27 Validation loss: 0.145124 Best loss: 0.055399 Accuracy: 98.40% 28 Validation loss: 0.143334 Best loss: 0.055399 Accuracy: 98.20% 29 Validation loss: 0.117147 Best loss: 0.055399 Accuracy: 98.75% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.05, total= 5.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.131452 Best loss: 0.131452 Accuracy: 96.44% 1 Validation loss: 0.080886 Best loss: 0.080886 Accuracy: 97.62% 2 Validation loss: 0.075743 Best loss: 0.075743 Accuracy: 97.81% 3 Validation loss: 0.071537 Best loss: 0.071537 Accuracy: 98.20% 4 Validation loss: 0.062232 Best loss: 0.062232 Accuracy: 98.16% 5 Validation loss: 0.058750 Best loss: 0.058750 Accuracy: 98.32% 6 Validation loss: 0.070599 Best loss: 0.058750 Accuracy: 97.77% 7 Validation loss: 0.057703 Best loss: 0.057703 Accuracy: 98.36% 8 Validation loss: 0.053053 Best loss: 0.053053 Accuracy: 98.55% 9 Validation loss: 0.068434 Best loss: 0.053053 Accuracy: 98.20% 10 Validation loss: 0.064046 Best loss: 0.053053 Accuracy: 98.36% 11 Validation loss: 0.057729 Best loss: 0.053053 Accuracy: 98.63% 12 Validation loss: 0.079719 Best loss: 0.053053 Accuracy: 98.16% 13 Validation loss: 0.075266 Best loss: 0.053053 Accuracy: 97.97% 14 Validation loss: 0.073842 Best loss: 0.053053 Accuracy: 98.51% 15 Validation loss: 0.052342 Best loss: 0.052342 Accuracy: 98.59% 16 Validation loss: 0.063442 Best loss: 0.052342 Accuracy: 98.36% 17 Validation loss: 0.219761 Best loss: 0.052342 Accuracy: 96.60% 18 Validation loss: 0.073877 Best loss: 0.052342 Accuracy: 98.08% 19 Validation loss: 0.060228 Best loss: 0.052342 Accuracy: 98.59% 20 Validation loss: 0.053642 Best loss: 0.052342 Accuracy: 98.67% 21 Validation loss: 0.067238 Best loss: 0.052342 Accuracy: 98.79% 22 Validation loss: 0.049928 Best loss: 0.049928 Accuracy: 98.71% 23 Validation loss: 0.055626 Best loss: 0.049928 Accuracy: 98.75% 24 Validation loss: 0.056890 Best loss: 0.049928 Accuracy: 98.55% 25 Validation loss: 0.079407 Best loss: 0.049928 Accuracy: 98.36% 26 Validation loss: 0.065610 Best loss: 0.049928 Accuracy: 98.59% 27 Validation loss: 0.062985 Best loss: 0.049928 Accuracy: 98.71% 28 Validation loss: 0.076539 Best loss: 0.049928 Accuracy: 98.44% 29 Validation loss: 0.062330 Best loss: 0.049928 Accuracy: 98.87% 30 Validation loss: 0.120019 Best loss: 0.049928 Accuracy: 97.97% 31 Validation loss: 0.122391 Best loss: 0.049928 Accuracy: 98.16% 32 Validation loss: 0.091533 Best loss: 0.049928 Accuracy: 98.51% 33 Validation loss: 0.091890 Best loss: 0.049928 Accuracy: 98.55% 34 Validation loss: 0.075114 Best loss: 0.049928 Accuracy: 98.83% 35 Validation loss: 0.072852 Best loss: 0.049928 Accuracy: 98.71% 36 Validation loss: 0.065643 Best loss: 0.049928 Accuracy: 98.98% 37 Validation loss: 0.127304 Best loss: 0.049928 Accuracy: 98.12% 38 Validation loss: 0.073570 Best loss: 0.049928 Accuracy: 98.67% 39 Validation loss: 0.069882 Best loss: 0.049928 Accuracy: 98.87% 40 Validation loss: 0.074950 Best loss: 0.049928 Accuracy: 98.59% 41 Validation loss: 0.074010 Best loss: 0.049928 Accuracy: 98.48% 42 Validation loss: 0.069592 Best loss: 0.049928 Accuracy: 98.71% 43 Validation loss: 0.070970 Best loss: 0.049928 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.8min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.106235 Best loss: 0.106235 Accuracy: 97.03% 1 Validation loss: 0.102443 Best loss: 0.102443 Accuracy: 97.15% 2 Validation loss: 0.106286 Best loss: 0.102443 Accuracy: 97.54% 3 Validation loss: 0.073298 Best loss: 0.073298 Accuracy: 98.08% 4 Validation loss: 0.086271 Best loss: 0.073298 Accuracy: 97.50% 5 Validation loss: 0.104406 Best loss: 0.073298 Accuracy: 97.62% 6 Validation loss: 0.092083 Best loss: 0.073298 Accuracy: 97.69% 7 Validation loss: 0.066662 Best loss: 0.066662 Accuracy: 98.20% 8 Validation loss: 0.086285 Best loss: 0.066662 Accuracy: 97.93% 9 Validation loss: 0.064563 Best loss: 0.064563 Accuracy: 98.48% 10 Validation loss: 0.101513 Best loss: 0.064563 Accuracy: 98.05% 11 Validation loss: 0.077836 Best loss: 0.064563 Accuracy: 98.28% 12 Validation loss: 0.076051 Best loss: 0.064563 Accuracy: 98.01% 13 Validation loss: 0.056714 Best loss: 0.056714 Accuracy: 98.55% 14 Validation loss: 0.069062 Best loss: 0.056714 Accuracy: 98.12% 15 Validation loss: 0.069022 Best loss: 0.056714 Accuracy: 98.36% 16 Validation loss: 0.117012 Best loss: 0.056714 Accuracy: 97.69% 17 Validation loss: 0.116769 Best loss: 0.056714 Accuracy: 97.50% 18 Validation loss: 0.051578 Best loss: 0.051578 Accuracy: 98.67% 19 Validation loss: 0.072966 Best loss: 0.051578 Accuracy: 98.63% 20 Validation loss: 0.056581 Best loss: 0.051578 Accuracy: 98.63% 21 Validation loss: 0.081095 Best loss: 0.051578 Accuracy: 98.32% 22 Validation loss: 0.081424 Best loss: 0.051578 Accuracy: 98.55% 23 Validation loss: 0.052348 Best loss: 0.051578 Accuracy: 98.71% 24 Validation loss: 0.058600 Best loss: 0.051578 Accuracy: 98.67% 25 Validation loss: 0.113429 Best loss: 0.051578 Accuracy: 98.36% 26 Validation loss: 0.105102 Best loss: 0.051578 Accuracy: 98.20% 27 Validation loss: 0.089076 Best loss: 0.051578 Accuracy: 98.63% 28 Validation loss: 0.119500 Best loss: 0.051578 Accuracy: 98.36% 29 Validation loss: 0.086452 Best loss: 0.051578 Accuracy: 98.44% 30 Validation loss: 0.064417 Best loss: 0.051578 Accuracy: 98.87% 31 Validation loss: 0.079650 Best loss: 0.051578 Accuracy: 98.71% 32 Validation loss: 0.062512 Best loss: 0.051578 Accuracy: 98.75% 33 Validation loss: 0.115293 Best loss: 0.051578 Accuracy: 98.59% 34 Validation loss: 0.111100 Best loss: 0.051578 Accuracy: 98.48% 35 Validation loss: 0.092442 Best loss: 0.051578 Accuracy: 98.79% 36 Validation loss: 0.115333 Best loss: 0.051578 Accuracy: 97.89% 37 Validation loss: 0.088129 Best loss: 0.051578 Accuracy: 98.67% 38 Validation loss: 0.669470 Best loss: 0.051578 Accuracy: 85.73% 39 Validation loss: 0.089519 Best loss: 0.051578 Accuracy: 98.71% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.7min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1 0 Validation loss: 0.162887 Best loss: 0.162887 Accuracy: 96.25% 1 Validation loss: 0.124119 Best loss: 0.124119 Accuracy: 96.64% 2 Validation loss: 0.075806 Best loss: 0.075806 Accuracy: 98.01% 3 Validation loss: 0.071734 Best loss: 0.071734 Accuracy: 98.08% 4 Validation loss: 0.063531 Best loss: 0.063531 Accuracy: 98.08% 5 Validation loss: 0.141545 Best loss: 0.063531 Accuracy: 96.25% 6 Validation loss: 0.087237 Best loss: 0.063531 Accuracy: 97.77% 7 Validation loss: 0.093520 Best loss: 0.063531 Accuracy: 98.20% 8 Validation loss: 0.052084 Best loss: 0.052084 Accuracy: 98.79% 9 Validation loss: 0.070176 Best loss: 0.052084 Accuracy: 98.32% 10 Validation loss: 0.057033 Best loss: 0.052084 Accuracy: 98.83% 11 Validation loss: 0.052245 Best loss: 0.052084 Accuracy: 98.83% 12 Validation loss: 0.084441 Best loss: 0.052084 Accuracy: 97.81% 13 Validation loss: 0.061655 Best loss: 0.052084 Accuracy: 98.36% 14 Validation loss: 0.071090 Best loss: 0.052084 Accuracy: 98.44% 15 Validation loss: 0.087937 Best loss: 0.052084 Accuracy: 98.40% 16 Validation loss: 0.075716 Best loss: 0.052084 Accuracy: 98.12% 17 Validation loss: 0.064654 Best loss: 0.052084 Accuracy: 98.40% 18 Validation loss: 0.228329 Best loss: 0.052084 Accuracy: 95.74% 19 Validation loss: 0.093422 Best loss: 0.052084 Accuracy: 98.59% 20 Validation loss: 0.073053 Best loss: 0.052084 Accuracy: 98.71% 21 Validation loss: 0.077636 Best loss: 0.052084 Accuracy: 98.48% 22 Validation loss: 0.046424 Best loss: 0.046424 Accuracy: 98.87% 23 Validation loss: 0.062185 Best loss: 0.046424 Accuracy: 98.67% 24 Validation loss: 0.240824 Best loss: 0.046424 Accuracy: 97.85% 25 Validation loss: 0.065604 Best loss: 0.046424 Accuracy: 98.75% 26 Validation loss: 0.067849 Best loss: 0.046424 Accuracy: 98.83% 27 Validation loss: 0.058649 Best loss: 0.046424 Accuracy: 98.83% 28 Validation loss: 0.060835 Best loss: 0.046424 Accuracy: 98.91% 29 Validation loss: 0.056934 Best loss: 0.046424 Accuracy: 98.87% 30 Validation loss: 0.103297 Best loss: 0.046424 Accuracy: 97.85% 31 Validation loss: 0.086920 Best loss: 0.046424 Accuracy: 98.48% 32 Validation loss: 0.095728 Best loss: 0.046424 Accuracy: 98.87% 33 Validation loss: 0.116924 Best loss: 0.046424 Accuracy: 98.01% 34 Validation loss: 0.080389 Best loss: 0.046424 Accuracy: 98.87% 35 Validation loss: 0.083670 Best loss: 0.046424 Accuracy: 98.91% 36 Validation loss: 0.079843 Best loss: 0.046424 Accuracy: 98.87% 37 Validation loss: 0.162436 Best loss: 0.046424 Accuracy: 98.63% 38 Validation loss: 0.108198 Best loss: 0.046424 Accuracy: 98.59% 39 Validation loss: 0.091056 Best loss: 0.046424 Accuracy: 98.91% 40 Validation loss: 0.072438 Best loss: 0.046424 Accuracy: 98.94% 41 Validation loss: 0.090022 Best loss: 0.046424 Accuracy: 99.02% 42 Validation loss: 0.084884 Best loss: 0.046424 Accuracy: 98.75% 43 Validation loss: 0.083166 Best loss: 0.046424 Accuracy: 98.59% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=70, batch_size=50, batch_norm_momentum=0.95, learning_rate=0.1, total= 1.8min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.239628 Best loss: 0.239628 Accuracy: 92.14% 1 Validation loss: 0.880355 Best loss: 0.239628 Accuracy: 76.66% 2 Validation loss: 0.260279 Best loss: 0.239628 Accuracy: 91.36% 3 Validation loss: 0.455855 Best loss: 0.239628 Accuracy: 87.72% 4 Validation loss: 0.840935 Best loss: 0.239628 Accuracy: 85.11% 5 Validation loss: 0.332757 Best loss: 0.239628 Accuracy: 92.22% 6 Validation loss: 0.291610 Best loss: 0.239628 Accuracy: 94.14% 7 Validation loss: 0.233958 Best loss: 0.233958 Accuracy: 95.74% 8 Validation loss: 0.243789 Best loss: 0.233958 Accuracy: 95.74% 9 Validation loss: 0.406132 Best loss: 0.233958 Accuracy: 93.63% 10 Validation loss: 0.219415 Best loss: 0.219415 Accuracy: 96.64% 11 Validation loss: 0.241077 Best loss: 0.219415 Accuracy: 96.79% 12 Validation loss: 0.212679 Best loss: 0.212679 Accuracy: 96.99% 13 Validation loss: 0.223811 Best loss: 0.212679 Accuracy: 97.07% 14 Validation loss: 0.149854 Best loss: 0.149854 Accuracy: 98.20% 15 Validation loss: 0.202995 Best loss: 0.149854 Accuracy: 97.97% 16 Validation loss: 0.287241 Best loss: 0.149854 Accuracy: 96.64% 17 Validation loss: 0.171764 Best loss: 0.149854 Accuracy: 98.16% 18 Validation loss: 0.202716 Best loss: 0.149854 Accuracy: 97.69% 19 Validation loss: 0.219326 Best loss: 0.149854 Accuracy: 97.65% 20 Validation loss: 0.239475 Best loss: 0.149854 Accuracy: 97.50% 21 Validation loss: 0.261643 Best loss: 0.149854 Accuracy: 97.85% 22 Validation loss: 0.214376 Best loss: 0.149854 Accuracy: 98.20% 23 Validation loss: 0.185950 Best loss: 0.149854 Accuracy: 98.44% 24 Validation loss: 0.176298 Best loss: 0.149854 Accuracy: 98.32% 25 Validation loss: 0.178173 Best loss: 0.149854 Accuracy: 98.32% 26 Validation loss: 0.172675 Best loss: 0.149854 Accuracy: 98.36% 27 Validation loss: 0.190354 Best loss: 0.149854 Accuracy: 98.28% 28 Validation loss: 0.144666 Best loss: 0.144666 Accuracy: 98.67% 29 Validation loss: 0.146370 Best loss: 0.144666 Accuracy: 98.36% 30 Validation loss: 0.135295 Best loss: 0.135295 Accuracy: 98.71% 31 Validation loss: 0.135817 Best loss: 0.135295 Accuracy: 98.59% 32 Validation loss: 0.123317 Best loss: 0.123317 Accuracy: 98.67% 33 Validation loss: 0.118402 Best loss: 0.118402 Accuracy: 98.63% 34 Validation loss: 0.115268 Best loss: 0.115268 Accuracy: 98.51% 35 Validation loss: 0.106413 Best loss: 0.106413 Accuracy: 98.63% 36 Validation loss: 0.105033 Best loss: 0.105033 Accuracy: 98.67% 37 Validation loss: 0.105644 Best loss: 0.105033 Accuracy: 98.67% 38 Validation loss: 0.104263 Best loss: 0.104263 Accuracy: 98.71% 39 Validation loss: 0.107074 Best loss: 0.104263 Accuracy: 98.55% 40 Validation loss: 0.098929 Best loss: 0.098929 Accuracy: 98.71% 41 Validation loss: 0.092838 Best loss: 0.092838 Accuracy: 98.79% 42 Validation loss: 0.090766 Best loss: 0.090766 Accuracy: 98.87% 43 Validation loss: 0.087235 Best loss: 0.087235 Accuracy: 98.91% 44 Validation loss: 0.086588 Best loss: 0.086588 Accuracy: 98.94% 45 Validation loss: 0.086559 Best loss: 0.086559 Accuracy: 98.94% 46 Validation loss: 0.085224 Best loss: 0.085224 Accuracy: 98.91% 47 Validation loss: 0.083200 Best loss: 0.083200 Accuracy: 98.83% 48 Validation loss: 0.082473 Best loss: 0.082473 Accuracy: 98.83% 49 Validation loss: 0.082518 Best loss: 0.082473 Accuracy: 98.87% 50 Validation loss: 0.083057 Best loss: 0.082473 Accuracy: 98.79% 51 Validation loss: 0.081050 Best loss: 0.081050 Accuracy: 98.83% 52 Validation loss: 0.079977 Best loss: 0.079977 Accuracy: 98.79% 53 Validation loss: 0.080419 Best loss: 0.079977 Accuracy: 98.75% 54 Validation loss: 0.080317 Best loss: 0.079977 Accuracy: 98.71% 55 Validation loss: 0.079915 Best loss: 0.079915 Accuracy: 98.75% 56 Validation loss: 0.078619 Best loss: 0.078619 Accuracy: 98.71% 57 Validation loss: 0.078274 Best loss: 0.078274 Accuracy: 98.75% 58 Validation loss: 0.076279 Best loss: 0.076279 Accuracy: 98.79% 59 Validation loss: 0.075631 Best loss: 0.075631 Accuracy: 98.75% 60 Validation loss: 0.083838 Best loss: 0.075631 Accuracy: 98.59% 61 Validation loss: 0.081900 Best loss: 0.075631 Accuracy: 98.79% 62 Validation loss: 0.078721 Best loss: 0.075631 Accuracy: 98.83% 63 Validation loss: 0.078022 Best loss: 0.075631 Accuracy: 98.83% 64 Validation loss: 0.076568 Best loss: 0.075631 Accuracy: 98.79% 65 Validation loss: 0.075655 Best loss: 0.075631 Accuracy: 98.79% 66 Validation loss: 0.074845 Best loss: 0.074845 Accuracy: 98.79% 67 Validation loss: 0.076290 Best loss: 0.074845 Accuracy: 98.79% 68 Validation loss: 0.076936 Best loss: 0.074845 Accuracy: 98.75% 69 Validation loss: 0.076254 Best loss: 0.074845 Accuracy: 98.75% 70 Validation loss: 0.075523 Best loss: 0.074845 Accuracy: 98.83% 71 Validation loss: 0.075204 Best loss: 0.074845 Accuracy: 98.83% 72 Validation loss: 0.074985 Best loss: 0.074845 Accuracy: 98.75% 73 Validation loss: 0.073941 Best loss: 0.073941 Accuracy: 98.79% 74 Validation loss: 0.072743 Best loss: 0.072743 Accuracy: 98.79% 75 Validation loss: 0.072887 Best loss: 0.072743 Accuracy: 98.79% 76 Validation loss: 0.072645 Best loss: 0.072645 Accuracy: 98.83% 77 Validation loss: 0.072235 Best loss: 0.072235 Accuracy: 98.83% 78 Validation loss: 0.072536 Best loss: 0.072235 Accuracy: 98.79% 79 Validation loss: 0.072963 Best loss: 0.072235 Accuracy: 98.79% 80 Validation loss: 0.072825 Best loss: 0.072235 Accuracy: 98.79% 81 Validation loss: 0.073036 Best loss: 0.072235 Accuracy: 98.79% 82 Validation loss: 0.074210 Best loss: 0.072235 Accuracy: 98.79% 83 Validation loss: 0.074182 Best loss: 0.072235 Accuracy: 98.79% 84 Validation loss: 0.073095 Best loss: 0.072235 Accuracy: 98.79% 85 Validation loss: 0.072556 Best loss: 0.072235 Accuracy: 98.79% 86 Validation loss: 0.071967 Best loss: 0.071967 Accuracy: 98.79% 87 Validation loss: 0.072642 Best loss: 0.071967 Accuracy: 98.83% 88 Validation loss: 0.071711 Best loss: 0.071711 Accuracy: 98.83% 89 Validation loss: 0.071991 Best loss: 0.071711 Accuracy: 98.83% 90 Validation loss: 0.071483 Best loss: 0.071483 Accuracy: 98.83% 91 Validation loss: 0.072639 Best loss: 0.071483 Accuracy: 98.83% 92 Validation loss: 0.073179 Best loss: 0.071483 Accuracy: 98.79% 93 Validation loss: 0.073669 Best loss: 0.071483 Accuracy: 98.75% 94 Validation loss: 0.072615 Best loss: 0.071483 Accuracy: 98.75% 95 Validation loss: 0.071685 Best loss: 0.071483 Accuracy: 98.71% 96 Validation loss: 0.071933 Best loss: 0.071483 Accuracy: 98.79% 97 Validation loss: 0.072064 Best loss: 0.071483 Accuracy: 98.79% 98 Validation loss: 0.071719 Best loss: 0.071483 Accuracy: 98.79% 99 Validation loss: 0.072976 Best loss: 0.071483 Accuracy: 98.75% 100 Validation loss: 0.074340 Best loss: 0.071483 Accuracy: 98.75% 101 Validation loss: 0.074061 Best loss: 0.071483 Accuracy: 98.79% 102 Validation loss: 0.073505 Best loss: 0.071483 Accuracy: 98.75% 103 Validation loss: 0.072565 Best loss: 0.071483 Accuracy: 98.75% 104 Validation loss: 0.072818 Best loss: 0.071483 Accuracy: 98.79% 105 Validation loss: 0.072395 Best loss: 0.071483 Accuracy: 98.75% 106 Validation loss: 0.072318 Best loss: 0.071483 Accuracy: 98.71% 107 Validation loss: 0.072013 Best loss: 0.071483 Accuracy: 98.71% 108 Validation loss: 0.072579 Best loss: 0.071483 Accuracy: 98.75% 109 Validation loss: 0.072948 Best loss: 0.071483 Accuracy: 98.75% 110 Validation loss: 0.074016 Best loss: 0.071483 Accuracy: 98.75% 111 Validation loss: 0.073645 Best loss: 0.071483 Accuracy: 98.67% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01, total= 36.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.247271 Best loss: 0.247271 Accuracy: 92.42% 1 Validation loss: 0.147588 Best loss: 0.147588 Accuracy: 95.62% 2 Validation loss: 0.109846 Best loss: 0.109846 Accuracy: 96.64% 3 Validation loss: 0.316879 Best loss: 0.109846 Accuracy: 91.91% 4 Validation loss: 0.190760 Best loss: 0.109846 Accuracy: 95.78% 5 Validation loss: 0.182995 Best loss: 0.109846 Accuracy: 96.44% 6 Validation loss: 0.091512 Best loss: 0.091512 Accuracy: 98.20% 7 Validation loss: 0.166966 Best loss: 0.091512 Accuracy: 96.60% 8 Validation loss: 0.271829 Best loss: 0.091512 Accuracy: 94.57% 9 Validation loss: 0.171425 Best loss: 0.091512 Accuracy: 97.34% 10 Validation loss: 0.138819 Best loss: 0.091512 Accuracy: 98.16% 11 Validation loss: 0.157447 Best loss: 0.091512 Accuracy: 97.93% 12 Validation loss: 0.212304 Best loss: 0.091512 Accuracy: 97.19% 13 Validation loss: 0.275911 Best loss: 0.091512 Accuracy: 96.21% 14 Validation loss: 0.169266 Best loss: 0.091512 Accuracy: 97.62% 15 Validation loss: 0.304671 Best loss: 0.091512 Accuracy: 96.72% 16 Validation loss: 0.285457 Best loss: 0.091512 Accuracy: 96.95% 17 Validation loss: 0.206968 Best loss: 0.091512 Accuracy: 97.77% 18 Validation loss: 0.181110 Best loss: 0.091512 Accuracy: 97.81% 19 Validation loss: 0.236823 Best loss: 0.091512 Accuracy: 97.81% 20 Validation loss: 0.211839 Best loss: 0.091512 Accuracy: 98.20% 21 Validation loss: 0.329648 Best loss: 0.091512 Accuracy: 96.87% 22 Validation loss: 0.192634 Best loss: 0.091512 Accuracy: 98.32% 23 Validation loss: 0.155813 Best loss: 0.091512 Accuracy: 98.51% 24 Validation loss: 0.197261 Best loss: 0.091512 Accuracy: 97.77% 25 Validation loss: 0.216640 Best loss: 0.091512 Accuracy: 97.42% 26 Validation loss: 0.260608 Best loss: 0.091512 Accuracy: 97.77% 27 Validation loss: 0.209938 Best loss: 0.091512 Accuracy: 98.12% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01, total= 9.4s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01 0 Validation loss: 0.301342 Best loss: 0.301342 Accuracy: 89.95% 1 Validation loss: 0.171250 Best loss: 0.171250 Accuracy: 94.80% 2 Validation loss: 0.296626 Best loss: 0.171250 Accuracy: 91.44% 3 Validation loss: 0.233421 Best loss: 0.171250 Accuracy: 93.75% 4 Validation loss: 0.391073 Best loss: 0.171250 Accuracy: 90.54% 5 Validation loss: 0.172467 Best loss: 0.171250 Accuracy: 95.78% 6 Validation loss: 0.143021 Best loss: 0.143021 Accuracy: 96.68% 7 Validation loss: 0.253753 Best loss: 0.143021 Accuracy: 93.71% 8 Validation loss: 0.177590 Best loss: 0.143021 Accuracy: 96.60% 9 Validation loss: 0.271350 Best loss: 0.143021 Accuracy: 94.41% 10 Validation loss: 0.163258 Best loss: 0.143021 Accuracy: 97.34% 11 Validation loss: 0.129052 Best loss: 0.129052 Accuracy: 97.89% 12 Validation loss: 0.735352 Best loss: 0.129052 Accuracy: 89.33% 13 Validation loss: 0.293567 Best loss: 0.129052 Accuracy: 95.66% 14 Validation loss: 0.198449 Best loss: 0.129052 Accuracy: 97.30% 15 Validation loss: 0.214206 Best loss: 0.129052 Accuracy: 96.99% 16 Validation loss: 0.389744 Best loss: 0.129052 Accuracy: 94.88% 17 Validation loss: 0.164684 Best loss: 0.129052 Accuracy: 98.59% 18 Validation loss: 0.158234 Best loss: 0.129052 Accuracy: 97.85% 19 Validation loss: 0.114335 Best loss: 0.114335 Accuracy: 98.91% 20 Validation loss: 0.214169 Best loss: 0.114335 Accuracy: 97.46% 21 Validation loss: 0.129071 Best loss: 0.114335 Accuracy: 98.63% 22 Validation loss: 0.156316 Best loss: 0.114335 Accuracy: 97.93% 23 Validation loss: 0.138948 Best loss: 0.114335 Accuracy: 98.36% 24 Validation loss: 0.258332 Best loss: 0.114335 Accuracy: 97.30% 25 Validation loss: 0.153512 Best loss: 0.114335 Accuracy: 98.48% 26 Validation loss: 0.211228 Best loss: 0.114335 Accuracy: 98.16% 27 Validation loss: 0.201054 Best loss: 0.114335 Accuracy: 98.12% 28 Validation loss: 0.199338 Best loss: 0.114335 Accuracy: 98.24% 29 Validation loss: 0.249054 Best loss: 0.114335 Accuracy: 98.05% 30 Validation loss: 0.246406 Best loss: 0.114335 Accuracy: 98.24% 31 Validation loss: 0.164093 Best loss: 0.114335 Accuracy: 98.51% 32 Validation loss: 0.166684 Best loss: 0.114335 Accuracy: 98.67% 33 Validation loss: 0.149731 Best loss: 0.114335 Accuracy: 98.87% 34 Validation loss: 0.135417 Best loss: 0.114335 Accuracy: 98.71% 35 Validation loss: 0.134735 Best loss: 0.114335 Accuracy: 98.71% 36 Validation loss: 0.118873 Best loss: 0.114335 Accuracy: 98.91% 37 Validation loss: 0.122616 Best loss: 0.114335 Accuracy: 98.91% 38 Validation loss: 0.105053 Best loss: 0.105053 Accuracy: 98.94% 39 Validation loss: 0.109054 Best loss: 0.105053 Accuracy: 98.87% 40 Validation loss: 0.098395 Best loss: 0.098395 Accuracy: 98.91% 41 Validation loss: 0.095331 Best loss: 0.095331 Accuracy: 98.91% 42 Validation loss: 0.096169 Best loss: 0.095331 Accuracy: 98.91% 43 Validation loss: 0.090207 Best loss: 0.090207 Accuracy: 98.79% 44 Validation loss: 0.096111 Best loss: 0.090207 Accuracy: 98.83% 45 Validation loss: 0.094400 Best loss: 0.090207 Accuracy: 98.91% 46 Validation loss: 0.091338 Best loss: 0.090207 Accuracy: 98.91% 47 Validation loss: 0.090896 Best loss: 0.090207 Accuracy: 98.83% 48 Validation loss: 0.090852 Best loss: 0.090207 Accuracy: 98.83% 49 Validation loss: 0.085336 Best loss: 0.085336 Accuracy: 98.83% 50 Validation loss: 0.082033 Best loss: 0.082033 Accuracy: 98.91% 51 Validation loss: 0.083496 Best loss: 0.082033 Accuracy: 98.94% 52 Validation loss: 0.088187 Best loss: 0.082033 Accuracy: 98.91% 53 Validation loss: 0.085752 Best loss: 0.082033 Accuracy: 98.91% 54 Validation loss: 0.083634 Best loss: 0.082033 Accuracy: 98.91% 55 Validation loss: 0.080820 Best loss: 0.080820 Accuracy: 98.91% 56 Validation loss: 0.079039 Best loss: 0.079039 Accuracy: 98.91% 57 Validation loss: 0.077102 Best loss: 0.077102 Accuracy: 98.94% 58 Validation loss: 0.077826 Best loss: 0.077102 Accuracy: 98.98% 59 Validation loss: 0.076818 Best loss: 0.076818 Accuracy: 98.98% 60 Validation loss: 0.076307 Best loss: 0.076307 Accuracy: 98.91% 61 Validation loss: 0.075552 Best loss: 0.075552 Accuracy: 98.94% 62 Validation loss: 0.074474 Best loss: 0.074474 Accuracy: 98.91% 63 Validation loss: 0.073191 Best loss: 0.073191 Accuracy: 98.91% 64 Validation loss: 0.072609 Best loss: 0.072609 Accuracy: 98.94% 65 Validation loss: 0.072008 Best loss: 0.072008 Accuracy: 98.94% 66 Validation loss: 0.071458 Best loss: 0.071458 Accuracy: 98.94% 67 Validation loss: 0.071377 Best loss: 0.071377 Accuracy: 98.94% 68 Validation loss: 0.070865 Best loss: 0.070865 Accuracy: 98.98% 69 Validation loss: 0.070851 Best loss: 0.070851 Accuracy: 98.98% 70 Validation loss: 0.070214 Best loss: 0.070214 Accuracy: 98.98% 71 Validation loss: 0.069621 Best loss: 0.069621 Accuracy: 99.02% 72 Validation loss: 0.068505 Best loss: 0.068505 Accuracy: 98.98% 73 Validation loss: 0.068146 Best loss: 0.068146 Accuracy: 98.98% 74 Validation loss: 0.068160 Best loss: 0.068146 Accuracy: 98.98% 75 Validation loss: 0.068397 Best loss: 0.068146 Accuracy: 98.94% 76 Validation loss: 0.068614 Best loss: 0.068146 Accuracy: 98.94% 77 Validation loss: 0.068697 Best loss: 0.068146 Accuracy: 98.94% 78 Validation loss: 0.068388 Best loss: 0.068146 Accuracy: 98.91% 79 Validation loss: 0.068879 Best loss: 0.068146 Accuracy: 98.91% 80 Validation loss: 0.068812 Best loss: 0.068146 Accuracy: 98.94% 81 Validation loss: 0.068151 Best loss: 0.068146 Accuracy: 98.98% 82 Validation loss: 0.067183 Best loss: 0.067183 Accuracy: 98.98% 83 Validation loss: 0.067071 Best loss: 0.067071 Accuracy: 98.94% 84 Validation loss: 0.068887 Best loss: 0.067071 Accuracy: 98.94% 85 Validation loss: 0.069295 Best loss: 0.067071 Accuracy: 98.94% 86 Validation loss: 0.068787 Best loss: 0.067071 Accuracy: 98.94% 87 Validation loss: 0.068289 Best loss: 0.067071 Accuracy: 98.94% 88 Validation loss: 0.067777 Best loss: 0.067071 Accuracy: 98.91% 89 Validation loss: 0.067549 Best loss: 0.067071 Accuracy: 98.91% 90 Validation loss: 0.068325 Best loss: 0.067071 Accuracy: 98.91% 91 Validation loss: 0.068989 Best loss: 0.067071 Accuracy: 98.91% 92 Validation loss: 0.068864 Best loss: 0.067071 Accuracy: 98.94% 93 Validation loss: 0.068368 Best loss: 0.067071 Accuracy: 98.91% 94 Validation loss: 0.068843 Best loss: 0.067071 Accuracy: 98.98% 95 Validation loss: 0.068626 Best loss: 0.067071 Accuracy: 98.94% 96 Validation loss: 0.068629 Best loss: 0.067071 Accuracy: 98.98% 97 Validation loss: 0.069266 Best loss: 0.067071 Accuracy: 98.94% 98 Validation loss: 0.069211 Best loss: 0.067071 Accuracy: 98.94% 99 Validation loss: 0.068287 Best loss: 0.067071 Accuracy: 98.94% 100 Validation loss: 0.068079 Best loss: 0.067071 Accuracy: 98.94% 101 Validation loss: 0.068769 Best loss: 0.067071 Accuracy: 98.94% 102 Validation loss: 0.069318 Best loss: 0.067071 Accuracy: 98.87% 103 Validation loss: 0.068459 Best loss: 0.067071 Accuracy: 98.87% 104 Validation loss: 0.068342 Best loss: 0.067071 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, batch_norm_momentum=0.999, learning_rate=0.01, total= 32.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02 0 Validation loss: 0.471024 Best loss: 0.471024 Accuracy: 95.54% 1 Validation loss: 0.149413 Best loss: 0.149413 Accuracy: 96.64% 2 Validation loss: 0.086382 Best loss: 0.086382 Accuracy: 97.89% 3 Validation loss: 0.073577 Best loss: 0.073577 Accuracy: 98.08% 4 Validation loss: 0.081966 Best loss: 0.073577 Accuracy: 97.07% 5 Validation loss: 0.143083 Best loss: 0.073577 Accuracy: 96.36% 6 Validation loss: 0.073449 Best loss: 0.073449 Accuracy: 97.62% 7 Validation loss: 0.067659 Best loss: 0.067659 Accuracy: 98.12% 8 Validation loss: 0.061535 Best loss: 0.061535 Accuracy: 98.24% 9 Validation loss: 0.053487 Best loss: 0.053487 Accuracy: 98.59% 10 Validation loss: 0.069634 Best loss: 0.053487 Accuracy: 98.16% 11 Validation loss: 0.064822 Best loss: 0.053487 Accuracy: 98.20% 12 Validation loss: 0.073191 Best loss: 0.053487 Accuracy: 98.51% 13 Validation loss: 0.046208 Best loss: 0.046208 Accuracy: 98.59% 14 Validation loss: 0.051892 Best loss: 0.046208 Accuracy: 98.55% 15 Validation loss: 0.042088 Best loss: 0.042088 Accuracy: 98.98% 16 Validation loss: 0.058136 Best loss: 0.042088 Accuracy: 98.24% 17 Validation loss: 0.041906 Best loss: 0.041906 Accuracy: 98.79% 18 Validation loss: 0.061408 Best loss: 0.041906 Accuracy: 98.24% 19 Validation loss: 0.048093 Best loss: 0.041906 Accuracy: 98.71% 20 Validation loss: 0.041956 Best loss: 0.041906 Accuracy: 98.83% 21 Validation loss: 0.044435 Best loss: 0.041906 Accuracy: 98.79% 22 Validation loss: 0.042317 Best loss: 0.041906 Accuracy: 98.79% 23 Validation loss: 0.048101 Best loss: 0.041906 Accuracy: 98.55% 24 Validation loss: 0.031114 Best loss: 0.031114 Accuracy: 99.06% 25 Validation loss: 0.066738 Best loss: 0.031114 Accuracy: 98.32% 26 Validation loss: 0.033332 Best loss: 0.031114 Accuracy: 99.14% 27 Validation loss: 0.046960 Best loss: 0.031114 Accuracy: 98.83% 28 Validation loss: 0.050430 Best loss: 0.031114 Accuracy: 98.83% 29 Validation loss: 0.069028 Best loss: 0.031114 Accuracy: 98.05% 30 Validation loss: 0.047668 Best loss: 0.031114 Accuracy: 98.55% 31 Validation loss: 0.064790 Best loss: 0.031114 Accuracy: 98.63% 32 Validation loss: 0.049807 Best loss: 0.031114 Accuracy: 98.87% 33 Validation loss: 0.044084 Best loss: 0.031114 Accuracy: 99.02% 34 Validation loss: 0.059224 Best loss: 0.031114 Accuracy: 98.75% 35 Validation loss: 0.084404 Best loss: 0.031114 Accuracy: 97.97% 36 Validation loss: 0.042528 Best loss: 0.031114 Accuracy: 98.87% 37 Validation loss: 0.047584 Best loss: 0.031114 Accuracy: 98.87% 38 Validation loss: 0.053518 Best loss: 0.031114 Accuracy: 98.94% 39 Validation loss: 0.043225 Best loss: 0.031114 Accuracy: 98.94% 40 Validation loss: 0.041136 Best loss: 0.031114 Accuracy: 99.10% 41 Validation loss: 0.061855 Best loss: 0.031114 Accuracy: 98.67% 42 Validation loss: 0.057801 Best loss: 0.031114 Accuracy: 98.79% 43 Validation loss: 0.058243 Best loss: 0.031114 Accuracy: 98.51% 44 Validation loss: 0.041648 Best loss: 0.031114 Accuracy: 98.79% 45 Validation loss: 0.038037 Best loss: 0.031114 Accuracy: 98.83% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02, total= 8.7min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02 0 Validation loss: 0.457487 Best loss: 0.457487 Accuracy: 94.68% 1 Validation loss: 0.237555 Best loss: 0.237555 Accuracy: 96.36% 2 Validation loss: 0.170453 Best loss: 0.170453 Accuracy: 96.36% 3 Validation loss: 0.097722 Best loss: 0.097722 Accuracy: 97.65% 4 Validation loss: 0.131439 Best loss: 0.097722 Accuracy: 96.99% 5 Validation loss: 0.082950 Best loss: 0.082950 Accuracy: 97.54% 6 Validation loss: 0.084688 Best loss: 0.082950 Accuracy: 97.69% 7 Validation loss: 0.078245 Best loss: 0.078245 Accuracy: 97.77% 8 Validation loss: 0.086665 Best loss: 0.078245 Accuracy: 97.30% 9 Validation loss: 0.069477 Best loss: 0.069477 Accuracy: 98.08% 10 Validation loss: 0.119247 Best loss: 0.069477 Accuracy: 96.68% 11 Validation loss: 0.056491 Best loss: 0.056491 Accuracy: 98.36% 12 Validation loss: 0.058942 Best loss: 0.056491 Accuracy: 98.40% 13 Validation loss: 0.059645 Best loss: 0.056491 Accuracy: 98.36% 14 Validation loss: 0.050912 Best loss: 0.050912 Accuracy: 98.12% 15 Validation loss: 0.091822 Best loss: 0.050912 Accuracy: 97.15% 16 Validation loss: 0.050514 Best loss: 0.050514 Accuracy: 98.59% 17 Validation loss: 0.049013 Best loss: 0.049013 Accuracy: 98.40% 18 Validation loss: 0.044196 Best loss: 0.044196 Accuracy: 98.63% 19 Validation loss: 0.049151 Best loss: 0.044196 Accuracy: 98.28% 20 Validation loss: 0.033328 Best loss: 0.033328 Accuracy: 98.83% 21 Validation loss: 0.038480 Best loss: 0.033328 Accuracy: 98.91% 22 Validation loss: 0.043915 Best loss: 0.033328 Accuracy: 98.63% 23 Validation loss: 0.039812 Best loss: 0.033328 Accuracy: 98.55% 24 Validation loss: 0.069546 Best loss: 0.033328 Accuracy: 98.05% 25 Validation loss: 0.066113 Best loss: 0.033328 Accuracy: 98.28% 26 Validation loss: 0.036323 Best loss: 0.033328 Accuracy: 98.87% 27 Validation loss: 0.060397 Best loss: 0.033328 Accuracy: 98.48% 28 Validation loss: 0.050727 Best loss: 0.033328 Accuracy: 98.55% 29 Validation loss: 0.044721 Best loss: 0.033328 Accuracy: 98.75% 30 Validation loss: 0.160953 Best loss: 0.033328 Accuracy: 95.74% 31 Validation loss: 0.037792 Best loss: 0.033328 Accuracy: 99.02% 32 Validation loss: 0.055541 Best loss: 0.033328 Accuracy: 98.51% 33 Validation loss: 0.043847 Best loss: 0.033328 Accuracy: 98.83% 34 Validation loss: 0.074334 Best loss: 0.033328 Accuracy: 98.05% 35 Validation loss: 0.035778 Best loss: 0.033328 Accuracy: 99.06% 36 Validation loss: 0.049639 Best loss: 0.033328 Accuracy: 98.40% 37 Validation loss: 0.044022 Best loss: 0.033328 Accuracy: 98.63% 38 Validation loss: 0.054284 Best loss: 0.033328 Accuracy: 98.48% 39 Validation loss: 0.048118 Best loss: 0.033328 Accuracy: 98.98% 40 Validation loss: 0.066846 Best loss: 0.033328 Accuracy: 98.48% 41 Validation loss: 0.045145 Best loss: 0.033328 Accuracy: 98.83% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02, total= 8.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02 0 Validation loss: 0.331754 Best loss: 0.331754 Accuracy: 96.13% 1 Validation loss: 0.120184 Best loss: 0.120184 Accuracy: 98.28% 2 Validation loss: 0.113179 Best loss: 0.113179 Accuracy: 97.77% 3 Validation loss: 0.060931 Best loss: 0.060931 Accuracy: 98.51% 4 Validation loss: 0.072612 Best loss: 0.060931 Accuracy: 98.12% 5 Validation loss: 0.124568 Best loss: 0.060931 Accuracy: 96.13% 6 Validation loss: 0.116638 Best loss: 0.060931 Accuracy: 96.79% 7 Validation loss: 0.051762 Best loss: 0.051762 Accuracy: 98.75% 8 Validation loss: 0.054486 Best loss: 0.051762 Accuracy: 98.40% 9 Validation loss: 0.063526 Best loss: 0.051762 Accuracy: 98.24% 10 Validation loss: 0.041350 Best loss: 0.041350 Accuracy: 98.55% 11 Validation loss: 0.062947 Best loss: 0.041350 Accuracy: 98.44% 12 Validation loss: 0.089592 Best loss: 0.041350 Accuracy: 97.77% 13 Validation loss: 0.088536 Best loss: 0.041350 Accuracy: 97.77% 14 Validation loss: 0.084755 Best loss: 0.041350 Accuracy: 97.58% 15 Validation loss: 0.055635 Best loss: 0.041350 Accuracy: 98.44% 16 Validation loss: 0.049642 Best loss: 0.041350 Accuracy: 98.59% 17 Validation loss: 0.033438 Best loss: 0.033438 Accuracy: 98.59% 18 Validation loss: 0.063930 Best loss: 0.033438 Accuracy: 98.36% 19 Validation loss: 0.047175 Best loss: 0.033438 Accuracy: 98.67% 20 Validation loss: 0.051331 Best loss: 0.033438 Accuracy: 98.91% 21 Validation loss: 0.036087 Best loss: 0.033438 Accuracy: 98.91% 22 Validation loss: 0.043372 Best loss: 0.033438 Accuracy: 98.67% 23 Validation loss: 0.039343 Best loss: 0.033438 Accuracy: 99.02% 24 Validation loss: 0.049003 Best loss: 0.033438 Accuracy: 98.79% 25 Validation loss: 0.057040 Best loss: 0.033438 Accuracy: 98.08% 26 Validation loss: 0.042359 Best loss: 0.033438 Accuracy: 98.87% 27 Validation loss: 0.070106 Best loss: 0.033438 Accuracy: 98.16% 28 Validation loss: 0.047427 Best loss: 0.033438 Accuracy: 98.71% 29 Validation loss: 0.039072 Best loss: 0.033438 Accuracy: 98.83% 30 Validation loss: 0.063050 Best loss: 0.033438 Accuracy: 98.59% 31 Validation loss: 0.040927 Best loss: 0.033438 Accuracy: 98.98% 32 Validation loss: 0.039054 Best loss: 0.033438 Accuracy: 98.91% 33 Validation loss: 0.048429 Best loss: 0.033438 Accuracy: 98.51% 34 Validation loss: 0.056309 Best loss: 0.033438 Accuracy: 98.79% 35 Validation loss: 0.066078 Best loss: 0.033438 Accuracy: 98.32% 36 Validation loss: 0.058672 Best loss: 0.033438 Accuracy: 98.51% 37 Validation loss: 0.051366 Best loss: 0.033438 Accuracy: 98.83% 38 Validation loss: 0.049801 Best loss: 0.033438 Accuracy: 98.67% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=90, batch_size=10, batch_norm_momentum=0.999, learning_rate=0.02, total= 7.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.083041 Best loss: 0.083041 Accuracy: 97.81% 1 Validation loss: 0.109134 Best loss: 0.083041 Accuracy: 97.11% 2 Validation loss: 0.090509 Best loss: 0.083041 Accuracy: 97.97% 3 Validation loss: 0.056629 Best loss: 0.056629 Accuracy: 98.40% 4 Validation loss: 0.049775 Best loss: 0.049775 Accuracy: 98.59% 5 Validation loss: 0.050430 Best loss: 0.049775 Accuracy: 98.79% 6 Validation loss: 0.048150 Best loss: 0.048150 Accuracy: 98.91% 7 Validation loss: 0.038431 Best loss: 0.038431 Accuracy: 98.75% 8 Validation loss: 0.101667 Best loss: 0.038431 Accuracy: 97.50% 9 Validation loss: 0.071999 Best loss: 0.038431 Accuracy: 98.51% 10 Validation loss: 0.031605 Best loss: 0.031605 Accuracy: 99.30% 11 Validation loss: 0.041497 Best loss: 0.031605 Accuracy: 98.83% 12 Validation loss: 0.043905 Best loss: 0.031605 Accuracy: 98.87% 13 Validation loss: 0.036159 Best loss: 0.031605 Accuracy: 99.22% 14 Validation loss: 0.060781 Best loss: 0.031605 Accuracy: 98.83% 15 Validation loss: 0.042726 Best loss: 0.031605 Accuracy: 99.14% 16 Validation loss: 0.051121 Best loss: 0.031605 Accuracy: 98.91% 17 Validation loss: 0.051902 Best loss: 0.031605 Accuracy: 98.94% 18 Validation loss: 0.059347 Best loss: 0.031605 Accuracy: 98.91% 19 Validation loss: 0.061768 Best loss: 0.031605 Accuracy: 98.63% 20 Validation loss: 0.043071 Best loss: 0.031605 Accuracy: 98.91% 21 Validation loss: 0.045696 Best loss: 0.031605 Accuracy: 98.87% 22 Validation loss: 0.043820 Best loss: 0.031605 Accuracy: 98.79% 23 Validation loss: 0.045215 Best loss: 0.031605 Accuracy: 98.87% 24 Validation loss: 0.033976 Best loss: 0.031605 Accuracy: 99.10% 25 Validation loss: 0.057044 Best loss: 0.031605 Accuracy: 98.79% 26 Validation loss: 0.037517 Best loss: 0.031605 Accuracy: 99.02% 27 Validation loss: 0.042015 Best loss: 0.031605 Accuracy: 98.98% 28 Validation loss: 0.044088 Best loss: 0.031605 Accuracy: 98.91% 29 Validation loss: 0.051447 Best loss: 0.031605 Accuracy: 98.98% 30 Validation loss: 0.053711 Best loss: 0.031605 Accuracy: 98.98% 31 Validation loss: 0.050767 Best loss: 0.031605 Accuracy: 98.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 45.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.064611 Best loss: 0.064611 Accuracy: 97.97% 1 Validation loss: 0.051374 Best loss: 0.051374 Accuracy: 98.51% 2 Validation loss: 0.057528 Best loss: 0.051374 Accuracy: 98.28% 3 Validation loss: 0.068829 Best loss: 0.051374 Accuracy: 98.12% 4 Validation loss: 0.084804 Best loss: 0.051374 Accuracy: 98.05% 5 Validation loss: 0.042733 Best loss: 0.042733 Accuracy: 98.75% 6 Validation loss: 0.044448 Best loss: 0.042733 Accuracy: 98.63% 7 Validation loss: 0.044311 Best loss: 0.042733 Accuracy: 98.79% 8 Validation loss: 0.061002 Best loss: 0.042733 Accuracy: 98.51% 9 Validation loss: 0.057136 Best loss: 0.042733 Accuracy: 98.63% 10 Validation loss: 0.065733 Best loss: 0.042733 Accuracy: 98.40% 11 Validation loss: 0.056811 Best loss: 0.042733 Accuracy: 98.79% 12 Validation loss: 0.041589 Best loss: 0.041589 Accuracy: 98.91% 13 Validation loss: 0.043691 Best loss: 0.041589 Accuracy: 98.94% 14 Validation loss: 0.069755 Best loss: 0.041589 Accuracy: 98.44% 15 Validation loss: 0.040930 Best loss: 0.040930 Accuracy: 98.94% 16 Validation loss: 0.041227 Best loss: 0.040930 Accuracy: 98.98% 17 Validation loss: 0.042633 Best loss: 0.040930 Accuracy: 99.06% 18 Validation loss: 0.051788 Best loss: 0.040930 Accuracy: 98.91% 19 Validation loss: 0.064752 Best loss: 0.040930 Accuracy: 98.55% 20 Validation loss: 0.047565 Best loss: 0.040930 Accuracy: 98.67% 21 Validation loss: 0.050159 Best loss: 0.040930 Accuracy: 98.75% 22 Validation loss: 0.046722 Best loss: 0.040930 Accuracy: 98.91% 23 Validation loss: 0.106500 Best loss: 0.040930 Accuracy: 98.16% 24 Validation loss: 0.042342 Best loss: 0.040930 Accuracy: 98.98% 25 Validation loss: 0.047868 Best loss: 0.040930 Accuracy: 98.83% 26 Validation loss: 0.055851 Best loss: 0.040930 Accuracy: 98.63% 27 Validation loss: 0.043390 Best loss: 0.040930 Accuracy: 99.18% 28 Validation loss: 0.057746 Best loss: 0.040930 Accuracy: 98.59% 29 Validation loss: 0.066710 Best loss: 0.040930 Accuracy: 98.75% 30 Validation loss: 0.061189 Best loss: 0.040930 Accuracy: 98.87% 31 Validation loss: 0.057716 Best loss: 0.040930 Accuracy: 99.02% 32 Validation loss: 0.048458 Best loss: 0.040930 Accuracy: 99.10% 33 Validation loss: 0.055408 Best loss: 0.040930 Accuracy: 98.98% 34 Validation loss: 0.091585 Best loss: 0.040930 Accuracy: 98.28% 35 Validation loss: 0.055077 Best loss: 0.040930 Accuracy: 98.83% 36 Validation loss: 0.064133 Best loss: 0.040930 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 53.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.081555 Best loss: 0.081555 Accuracy: 97.46% 1 Validation loss: 0.056999 Best loss: 0.056999 Accuracy: 98.28% 2 Validation loss: 0.089829 Best loss: 0.056999 Accuracy: 97.62% 3 Validation loss: 0.051981 Best loss: 0.051981 Accuracy: 98.32% 4 Validation loss: 0.048045 Best loss: 0.048045 Accuracy: 98.87% 5 Validation loss: 0.042116 Best loss: 0.042116 Accuracy: 98.79% 6 Validation loss: 0.044730 Best loss: 0.042116 Accuracy: 98.55% 7 Validation loss: 0.061705 Best loss: 0.042116 Accuracy: 98.55% 8 Validation loss: 0.052582 Best loss: 0.042116 Accuracy: 98.79% 9 Validation loss: 0.049591 Best loss: 0.042116 Accuracy: 99.10% 10 Validation loss: 0.049337 Best loss: 0.042116 Accuracy: 98.71% 11 Validation loss: 0.072595 Best loss: 0.042116 Accuracy: 98.44% 12 Validation loss: 0.031153 Best loss: 0.031153 Accuracy: 99.18% 13 Validation loss: 0.050284 Best loss: 0.031153 Accuracy: 98.98% 14 Validation loss: 0.033644 Best loss: 0.031153 Accuracy: 99.22% 15 Validation loss: 0.037652 Best loss: 0.031153 Accuracy: 99.22% 16 Validation loss: 0.038498 Best loss: 0.031153 Accuracy: 99.02% 17 Validation loss: 0.047961 Best loss: 0.031153 Accuracy: 98.87% 18 Validation loss: 0.052761 Best loss: 0.031153 Accuracy: 98.83% 19 Validation loss: 0.053056 Best loss: 0.031153 Accuracy: 98.91% 20 Validation loss: 0.047868 Best loss: 0.031153 Accuracy: 99.02% 21 Validation loss: 0.029875 Best loss: 0.029875 Accuracy: 99.26% 22 Validation loss: 0.043827 Best loss: 0.029875 Accuracy: 99.10% 23 Validation loss: 0.037889 Best loss: 0.029875 Accuracy: 99.02% 24 Validation loss: 0.040635 Best loss: 0.029875 Accuracy: 99.22% 25 Validation loss: 0.036723 Best loss: 0.029875 Accuracy: 99.14% 26 Validation loss: 0.040889 Best loss: 0.029875 Accuracy: 99.06% 27 Validation loss: 0.046213 Best loss: 0.029875 Accuracy: 98.94% 28 Validation loss: 0.031750 Best loss: 0.029875 Accuracy: 99.41% 29 Validation loss: 0.041382 Best loss: 0.029875 Accuracy: 98.98% 30 Validation loss: 0.036891 Best loss: 0.029875 Accuracy: 99.10% 31 Validation loss: 0.032851 Best loss: 0.029875 Accuracy: 99.30% 32 Validation loss: 0.038436 Best loss: 0.029875 Accuracy: 99.22% 33 Validation loss: 0.033033 Best loss: 0.029875 Accuracy: 99.22% 34 Validation loss: 0.057528 Best loss: 0.029875 Accuracy: 98.94% 35 Validation loss: 0.037729 Best loss: 0.029875 Accuracy: 99.26% 36 Validation loss: 0.043924 Best loss: 0.029875 Accuracy: 98.91% 37 Validation loss: 0.035814 Best loss: 0.029875 Accuracy: 99.18% 38 Validation loss: 0.040242 Best loss: 0.029875 Accuracy: 99.06% 39 Validation loss: 0.042204 Best loss: 0.029875 Accuracy: 99.02% 40 Validation loss: 0.030387 Best loss: 0.029875 Accuracy: 99.14% 41 Validation loss: 0.028905 Best loss: 0.028905 Accuracy: 99.22% 42 Validation loss: 0.043390 Best loss: 0.028905 Accuracy: 99.18% 43 Validation loss: 0.056300 Best loss: 0.028905 Accuracy: 99.02% 44 Validation loss: 0.037841 Best loss: 0.028905 Accuracy: 99.37% 45 Validation loss: 0.040991 Best loss: 0.028905 Accuracy: 99.26% 46 Validation loss: 0.044314 Best loss: 0.028905 Accuracy: 99.10% 47 Validation loss: 0.036000 Best loss: 0.028905 Accuracy: 99.22% 48 Validation loss: 0.046473 Best loss: 0.028905 Accuracy: 99.10% 49 Validation loss: 0.050611 Best loss: 0.028905 Accuracy: 99.06% 50 Validation loss: 0.051959 Best loss: 0.028905 Accuracy: 98.91% 51 Validation loss: 0.034973 Best loss: 0.028905 Accuracy: 99.18% 52 Validation loss: 0.036995 Best loss: 0.028905 Accuracy: 99.26% 53 Validation loss: 0.042857 Best loss: 0.028905 Accuracy: 99.06% 54 Validation loss: 0.044392 Best loss: 0.028905 Accuracy: 99.26% 55 Validation loss: 0.027918 Best loss: 0.027918 Accuracy: 99.41% 56 Validation loss: 0.033284 Best loss: 0.027918 Accuracy: 99.26% 57 Validation loss: 0.035315 Best loss: 0.027918 Accuracy: 99.18% 58 Validation loss: 0.028789 Best loss: 0.027918 Accuracy: 99.34% 59 Validation loss: 0.037837 Best loss: 0.027918 Accuracy: 99.22% 60 Validation loss: 0.036873 Best loss: 0.027918 Accuracy: 99.18% 61 Validation loss: 0.030670 Best loss: 0.027918 Accuracy: 99.37% 62 Validation loss: 0.037693 Best loss: 0.027918 Accuracy: 99.18% 63 Validation loss: 0.039109 Best loss: 0.027918 Accuracy: 99.26% 64 Validation loss: 0.050160 Best loss: 0.027918 Accuracy: 99.14% 65 Validation loss: 0.047263 Best loss: 0.027918 Accuracy: 99.06% 66 Validation loss: 0.034837 Best loss: 0.027918 Accuracy: 99.22% 67 Validation loss: 0.036156 Best loss: 0.027918 Accuracy: 99.10% 68 Validation loss: 0.043966 Best loss: 0.027918 Accuracy: 99.14% 69 Validation loss: 0.037817 Best loss: 0.027918 Accuracy: 99.14% 70 Validation loss: 0.032948 Best loss: 0.027918 Accuracy: 99.26% 71 Validation loss: 0.047844 Best loss: 0.027918 Accuracy: 99.06% 72 Validation loss: 0.033413 Best loss: 0.027918 Accuracy: 99.26% 73 Validation loss: 0.037859 Best loss: 0.027918 Accuracy: 99.18% 74 Validation loss: 0.041735 Best loss: 0.027918 Accuracy: 99.10% 75 Validation loss: 0.040344 Best loss: 0.027918 Accuracy: 99.26% 76 Validation loss: 0.039365 Best loss: 0.027918 Accuracy: 99.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, n_neurons=70, batch_size=100, batch_norm_momentum=0.98, learning_rate=0.01, total= 1.8min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.135895 Best loss: 0.135895 Accuracy: 95.93% 1 Validation loss: 0.089059 Best loss: 0.089059 Accuracy: 97.30% 2 Validation loss: 0.057538 Best loss: 0.057538 Accuracy: 98.28% 3 Validation loss: 0.056248 Best loss: 0.056248 Accuracy: 98.48% 4 Validation loss: 0.041040 Best loss: 0.041040 Accuracy: 98.91% 5 Validation loss: 0.053587 Best loss: 0.041040 Accuracy: 98.44% 6 Validation loss: 0.058413 Best loss: 0.041040 Accuracy: 98.20% 7 Validation loss: 0.040944 Best loss: 0.040944 Accuracy: 98.79% 8 Validation loss: 0.044487 Best loss: 0.040944 Accuracy: 98.75% 9 Validation loss: 0.048029 Best loss: 0.040944 Accuracy: 98.67% 10 Validation loss: 0.034603 Best loss: 0.034603 Accuracy: 98.87% 11 Validation loss: 0.031578 Best loss: 0.031578 Accuracy: 98.87% 12 Validation loss: 0.040314 Best loss: 0.031578 Accuracy: 98.94% 13 Validation loss: 0.057702 Best loss: 0.031578 Accuracy: 98.12% 14 Validation loss: 0.043874 Best loss: 0.031578 Accuracy: 98.71% 15 Validation loss: 0.033058 Best loss: 0.031578 Accuracy: 98.94% 16 Validation loss: 0.048768 Best loss: 0.031578 Accuracy: 98.71% 17 Validation loss: 0.028382 Best loss: 0.028382 Accuracy: 99.06% 18 Validation loss: 0.032824 Best loss: 0.028382 Accuracy: 99.10% 19 Validation loss: 0.037894 Best loss: 0.028382 Accuracy: 99.02% 20 Validation loss: 0.034805 Best loss: 0.028382 Accuracy: 98.94% 21 Validation loss: 0.044506 Best loss: 0.028382 Accuracy: 98.94% 22 Validation loss: 0.033534 Best loss: 0.028382 Accuracy: 99.14% 23 Validation loss: 0.035920 Best loss: 0.028382 Accuracy: 98.94% 24 Validation loss: 0.049116 Best loss: 0.028382 Accuracy: 98.63% 25 Validation loss: 0.030372 Best loss: 0.028382 Accuracy: 99.06% 26 Validation loss: 0.040390 Best loss: 0.028382 Accuracy: 98.79% 27 Validation loss: 0.029639 Best loss: 0.028382 Accuracy: 99.02% 28 Validation loss: 0.033928 Best loss: 0.028382 Accuracy: 99.30% 29 Validation loss: 0.032589 Best loss: 0.028382 Accuracy: 99.18% 30 Validation loss: 0.043415 Best loss: 0.028382 Accuracy: 98.94% 31 Validation loss: 0.037543 Best loss: 0.028382 Accuracy: 99.06% 32 Validation loss: 0.036452 Best loss: 0.028382 Accuracy: 98.91% 33 Validation loss: 0.050811 Best loss: 0.028382 Accuracy: 98.48% 34 Validation loss: 0.031195 Best loss: 0.028382 Accuracy: 99.06% 35 Validation loss: 0.041725 Best loss: 0.028382 Accuracy: 99.02% 36 Validation loss: 0.037258 Best loss: 0.028382 Accuracy: 98.87% 37 Validation loss: 0.042537 Best loss: 0.028382 Accuracy: 98.98% 38 Validation loss: 0.030796 Best loss: 0.028382 Accuracy: 99.14% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total= 6.6min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.113897 Best loss: 0.113897 Accuracy: 96.36% 1 Validation loss: 0.065825 Best loss: 0.065825 Accuracy: 98.12% 2 Validation loss: 0.068945 Best loss: 0.065825 Accuracy: 97.69% 3 Validation loss: 0.057080 Best loss: 0.057080 Accuracy: 98.32% 4 Validation loss: 0.046359 Best loss: 0.046359 Accuracy: 98.44% 5 Validation loss: 0.044383 Best loss: 0.044383 Accuracy: 98.63% 6 Validation loss: 0.043654 Best loss: 0.043654 Accuracy: 98.55% 7 Validation loss: 0.050287 Best loss: 0.043654 Accuracy: 98.48% 8 Validation loss: 0.036835 Best loss: 0.036835 Accuracy: 98.83% 9 Validation loss: 0.044200 Best loss: 0.036835 Accuracy: 98.44% 10 Validation loss: 0.042856 Best loss: 0.036835 Accuracy: 98.63% 11 Validation loss: 0.034719 Best loss: 0.034719 Accuracy: 98.83% 12 Validation loss: 0.039801 Best loss: 0.034719 Accuracy: 98.51% 13 Validation loss: 0.042361 Best loss: 0.034719 Accuracy: 98.91% 14 Validation loss: 0.035580 Best loss: 0.034719 Accuracy: 99.02% 15 Validation loss: 0.042268 Best loss: 0.034719 Accuracy: 98.71% 16 Validation loss: 0.042360 Best loss: 0.034719 Accuracy: 98.91% 17 Validation loss: 0.031786 Best loss: 0.031786 Accuracy: 98.91% 18 Validation loss: 0.037510 Best loss: 0.031786 Accuracy: 98.91% 19 Validation loss: 0.043132 Best loss: 0.031786 Accuracy: 98.79% 20 Validation loss: 0.036768 Best loss: 0.031786 Accuracy: 99.06% 21 Validation loss: 0.033432 Best loss: 0.031786 Accuracy: 99.06% 22 Validation loss: 0.045360 Best loss: 0.031786 Accuracy: 98.98% 23 Validation loss: 0.045963 Best loss: 0.031786 Accuracy: 98.87% 24 Validation loss: 0.037995 Best loss: 0.031786 Accuracy: 98.94% 25 Validation loss: 0.033006 Best loss: 0.031786 Accuracy: 98.91% 26 Validation loss: 0.028463 Best loss: 0.028463 Accuracy: 99.10% 27 Validation loss: 0.037093 Best loss: 0.028463 Accuracy: 98.79% 28 Validation loss: 0.039884 Best loss: 0.028463 Accuracy: 98.98% 29 Validation loss: 0.036482 Best loss: 0.028463 Accuracy: 99.10% 30 Validation loss: 0.038566 Best loss: 0.028463 Accuracy: 98.87% 31 Validation loss: 0.029829 Best loss: 0.028463 Accuracy: 99.10% 32 Validation loss: 0.038524 Best loss: 0.028463 Accuracy: 99.02% 33 Validation loss: 0.033066 Best loss: 0.028463 Accuracy: 99.30% 34 Validation loss: 0.035800 Best loss: 0.028463 Accuracy: 99.14% 35 Validation loss: 0.039593 Best loss: 0.028463 Accuracy: 98.75% 36 Validation loss: 0.029419 Best loss: 0.028463 Accuracy: 99.14% 37 Validation loss: 0.029986 Best loss: 0.028463 Accuracy: 99.26% 38 Validation loss: 0.045232 Best loss: 0.028463 Accuracy: 98.75% 39 Validation loss: 0.034207 Best loss: 0.028463 Accuracy: 99.10% 40 Validation loss: 0.030828 Best loss: 0.028463 Accuracy: 99.10% 41 Validation loss: 0.033682 Best loss: 0.028463 Accuracy: 99.06% 42 Validation loss: 0.030200 Best loss: 0.028463 Accuracy: 99.34% 43 Validation loss: 0.035797 Best loss: 0.028463 Accuracy: 99.10% 44 Validation loss: 0.031675 Best loss: 0.028463 Accuracy: 99.26% 45 Validation loss: 0.032206 Best loss: 0.028463 Accuracy: 99.22% 46 Validation loss: 0.041134 Best loss: 0.028463 Accuracy: 99.10% 47 Validation loss: 0.041653 Best loss: 0.028463 Accuracy: 99.14% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total= 8.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01 0 Validation loss: 0.088633 Best loss: 0.088633 Accuracy: 97.89% 1 Validation loss: 0.065723 Best loss: 0.065723 Accuracy: 98.20% 2 Validation loss: 0.064454 Best loss: 0.064454 Accuracy: 98.20% 3 Validation loss: 0.059132 Best loss: 0.059132 Accuracy: 98.48% 4 Validation loss: 0.058894 Best loss: 0.058894 Accuracy: 98.08% 5 Validation loss: 0.042323 Best loss: 0.042323 Accuracy: 98.83% 6 Validation loss: 0.069554 Best loss: 0.042323 Accuracy: 97.58% 7 Validation loss: 0.039287 Best loss: 0.039287 Accuracy: 98.98% 8 Validation loss: 0.042305 Best loss: 0.039287 Accuracy: 98.75% 9 Validation loss: 0.043414 Best loss: 0.039287 Accuracy: 98.59% 10 Validation loss: 0.044629 Best loss: 0.039287 Accuracy: 98.94% 11 Validation loss: 0.033023 Best loss: 0.033023 Accuracy: 99.06% 12 Validation loss: 0.032438 Best loss: 0.032438 Accuracy: 99.10% 13 Validation loss: 0.031507 Best loss: 0.031507 Accuracy: 99.06% 14 Validation loss: 0.039220 Best loss: 0.031507 Accuracy: 98.71% 15 Validation loss: 0.033846 Best loss: 0.031507 Accuracy: 98.94% 16 Validation loss: 0.033258 Best loss: 0.031507 Accuracy: 99.02% 17 Validation loss: 0.029112 Best loss: 0.029112 Accuracy: 99.22% 18 Validation loss: 0.034185 Best loss: 0.029112 Accuracy: 99.06% 19 Validation loss: 0.028136 Best loss: 0.028136 Accuracy: 98.94% 20 Validation loss: 0.037802 Best loss: 0.028136 Accuracy: 99.02% 21 Validation loss: 0.033368 Best loss: 0.028136 Accuracy: 99.10% 22 Validation loss: 0.035646 Best loss: 0.028136 Accuracy: 99.06% 23 Validation loss: 0.031120 Best loss: 0.028136 Accuracy: 99.02% 24 Validation loss: 0.033468 Best loss: 0.028136 Accuracy: 98.98% 25 Validation loss: 0.034383 Best loss: 0.028136 Accuracy: 98.94% 26 Validation loss: 0.035464 Best loss: 0.028136 Accuracy: 99.02% 27 Validation loss: 0.039122 Best loss: 0.028136 Accuracy: 98.79% 28 Validation loss: 0.029841 Best loss: 0.028136 Accuracy: 98.91% 29 Validation loss: 0.037994 Best loss: 0.028136 Accuracy: 98.79% 30 Validation loss: 0.031281 Best loss: 0.028136 Accuracy: 99.26% 31 Validation loss: 0.027955 Best loss: 0.027955 Accuracy: 99.22% 32 Validation loss: 0.027264 Best loss: 0.027264 Accuracy: 99.34% 33 Validation loss: 0.033202 Best loss: 0.027264 Accuracy: 99.22% 34 Validation loss: 0.029033 Best loss: 0.027264 Accuracy: 99.18% 35 Validation loss: 0.035187 Best loss: 0.027264 Accuracy: 99.10% 36 Validation loss: 0.030089 Best loss: 0.027264 Accuracy: 99.18% 37 Validation loss: 0.033785 Best loss: 0.027264 Accuracy: 99.18% 38 Validation loss: 0.035322 Best loss: 0.027264 Accuracy: 99.22% 39 Validation loss: 0.032870 Best loss: 0.027264 Accuracy: 99.02% 40 Validation loss: 0.031576 Best loss: 0.027264 Accuracy: 98.98% 41 Validation loss: 0.037053 Best loss: 0.027264 Accuracy: 99.10% 42 Validation loss: 0.037183 Best loss: 0.027264 Accuracy: 99.02% 43 Validation loss: 0.031911 Best loss: 0.027264 Accuracy: 99.14% 44 Validation loss: 0.035495 Best loss: 0.027264 Accuracy: 99.06% 45 Validation loss: 0.042840 Best loss: 0.027264 Accuracy: 98.87% 46 Validation loss: 0.039086 Best loss: 0.027264 Accuracy: 99.02% 47 Validation loss: 0.031523 Best loss: 0.027264 Accuracy: 99.10% 48 Validation loss: 0.047983 Best loss: 0.027264 Accuracy: 98.83% 49 Validation loss: 0.035494 Best loss: 0.027264 Accuracy: 99.02% 50 Validation loss: 0.032825 Best loss: 0.027264 Accuracy: 99.22% 51 Validation loss: 0.035362 Best loss: 0.027264 Accuracy: 99.18% 52 Validation loss: 0.037776 Best loss: 0.027264 Accuracy: 99.14% 53 Validation loss: 0.031994 Best loss: 0.027264 Accuracy: 99.18% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=10, batch_norm_momentum=0.98, learning_rate=0.01, total= 9.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.181980 Best loss: 0.181980 Accuracy: 96.64% 1 Validation loss: 0.110109 Best loss: 0.110109 Accuracy: 97.73% 2 Validation loss: 0.144208 Best loss: 0.110109 Accuracy: 97.07% 3 Validation loss: 0.102023 Best loss: 0.102023 Accuracy: 97.69% 4 Validation loss: 0.199664 Best loss: 0.102023 Accuracy: 96.56% 5 Validation loss: 0.088860 Best loss: 0.088860 Accuracy: 98.01% 6 Validation loss: 0.062041 Best loss: 0.062041 Accuracy: 98.36% 7 Validation loss: 0.066985 Best loss: 0.062041 Accuracy: 98.75% 8 Validation loss: 0.113112 Best loss: 0.062041 Accuracy: 97.73% 9 Validation loss: 0.079269 Best loss: 0.062041 Accuracy: 98.36% 10 Validation loss: 0.058212 Best loss: 0.058212 Accuracy: 98.75% 11 Validation loss: 0.109300 Best loss: 0.058212 Accuracy: 97.93% 12 Validation loss: 0.049217 Best loss: 0.049217 Accuracy: 98.91% 13 Validation loss: 0.043191 Best loss: 0.043191 Accuracy: 99.06% 14 Validation loss: 0.110122 Best loss: 0.043191 Accuracy: 97.81% 15 Validation loss: 0.123642 Best loss: 0.043191 Accuracy: 98.12% 16 Validation loss: 0.058975 Best loss: 0.043191 Accuracy: 98.79% 17 Validation loss: 0.095477 Best loss: 0.043191 Accuracy: 98.67% 18 Validation loss: 0.124868 Best loss: 0.043191 Accuracy: 97.77% 19 Validation loss: 0.073419 Best loss: 0.043191 Accuracy: 98.98% 20 Validation loss: 0.054654 Best loss: 0.043191 Accuracy: 99.10% 21 Validation loss: 0.091604 Best loss: 0.043191 Accuracy: 98.51% 22 Validation loss: 0.076160 Best loss: 0.043191 Accuracy: 98.87% 23 Validation loss: 0.076944 Best loss: 0.043191 Accuracy: 98.48% 24 Validation loss: 0.070008 Best loss: 0.043191 Accuracy: 98.67% 25 Validation loss: 0.036811 Best loss: 0.036811 Accuracy: 99.30% 26 Validation loss: 0.043631 Best loss: 0.036811 Accuracy: 99.06% 27 Validation loss: 0.122937 Best loss: 0.036811 Accuracy: 98.40% 28 Validation loss: 0.111995 Best loss: 0.036811 Accuracy: 98.51% 29 Validation loss: 0.051839 Best loss: 0.036811 Accuracy: 99.06% 30 Validation loss: 0.091498 Best loss: 0.036811 Accuracy: 98.32% 31 Validation loss: 0.106067 Best loss: 0.036811 Accuracy: 98.63% 32 Validation loss: 0.065469 Best loss: 0.036811 Accuracy: 98.59% 33 Validation loss: 0.132759 Best loss: 0.036811 Accuracy: 98.28% 34 Validation loss: 0.076360 Best loss: 0.036811 Accuracy: 98.67% 35 Validation loss: 0.061318 Best loss: 0.036811 Accuracy: 98.87% 36 Validation loss: 0.055328 Best loss: 0.036811 Accuracy: 98.87% 37 Validation loss: 0.112697 Best loss: 0.036811 Accuracy: 98.67% 38 Validation loss: 0.137597 Best loss: 0.036811 Accuracy: 98.16% 39 Validation loss: 0.065435 Best loss: 0.036811 Accuracy: 98.83% 40 Validation loss: 0.081353 Best loss: 0.036811 Accuracy: 98.67% 41 Validation loss: 0.116500 Best loss: 0.036811 Accuracy: 98.32% 42 Validation loss: 0.138083 Best loss: 0.036811 Accuracy: 98.44% 43 Validation loss: 0.174829 Best loss: 0.036811 Accuracy: 98.67% 44 Validation loss: 0.102646 Best loss: 0.036811 Accuracy: 98.67% 45 Validation loss: 0.071730 Best loss: 0.036811 Accuracy: 98.87% 46 Validation loss: 0.088371 Best loss: 0.036811 Accuracy: 98.67% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 1.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.242851 Best loss: 0.242851 Accuracy: 96.05% 1 Validation loss: 0.120136 Best loss: 0.120136 Accuracy: 97.30% 2 Validation loss: 0.213439 Best loss: 0.120136 Accuracy: 95.86% 3 Validation loss: 0.107910 Best loss: 0.107910 Accuracy: 97.58% 4 Validation loss: 0.075730 Best loss: 0.075730 Accuracy: 98.16% 5 Validation loss: 0.073752 Best loss: 0.073752 Accuracy: 98.40% 6 Validation loss: 0.131514 Best loss: 0.073752 Accuracy: 97.65% 7 Validation loss: 0.080015 Best loss: 0.073752 Accuracy: 97.89% 8 Validation loss: 0.044797 Best loss: 0.044797 Accuracy: 98.51% 9 Validation loss: 0.073187 Best loss: 0.044797 Accuracy: 98.71% 10 Validation loss: 0.103641 Best loss: 0.044797 Accuracy: 98.20% 11 Validation loss: 0.073847 Best loss: 0.044797 Accuracy: 98.59% 12 Validation loss: 0.061817 Best loss: 0.044797 Accuracy: 98.63% 13 Validation loss: 0.063837 Best loss: 0.044797 Accuracy: 98.67% 14 Validation loss: 0.078681 Best loss: 0.044797 Accuracy: 98.48% 15 Validation loss: 0.046345 Best loss: 0.044797 Accuracy: 98.67% 16 Validation loss: 0.048865 Best loss: 0.044797 Accuracy: 98.94% 17 Validation loss: 0.754683 Best loss: 0.044797 Accuracy: 94.88% 18 Validation loss: 0.100068 Best loss: 0.044797 Accuracy: 97.81% 19 Validation loss: 0.061732 Best loss: 0.044797 Accuracy: 99.06% 20 Validation loss: 0.120683 Best loss: 0.044797 Accuracy: 97.89% 21 Validation loss: 0.053226 Best loss: 0.044797 Accuracy: 98.75% 22 Validation loss: 0.176157 Best loss: 0.044797 Accuracy: 96.60% 23 Validation loss: 0.133085 Best loss: 0.044797 Accuracy: 98.44% 24 Validation loss: 0.116742 Best loss: 0.044797 Accuracy: 98.36% 25 Validation loss: 0.081468 Best loss: 0.044797 Accuracy: 98.67% 26 Validation loss: 0.065100 Best loss: 0.044797 Accuracy: 98.91% 27 Validation loss: 0.057281 Best loss: 0.044797 Accuracy: 99.02% 28 Validation loss: 0.280055 Best loss: 0.044797 Accuracy: 97.65% 29 Validation loss: 0.141887 Best loss: 0.044797 Accuracy: 97.97% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 41.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1 0 Validation loss: 0.217829 Best loss: 0.217829 Accuracy: 96.21% 1 Validation loss: 0.137937 Best loss: 0.137937 Accuracy: 97.69% 2 Validation loss: 0.120555 Best loss: 0.120555 Accuracy: 97.62% 3 Validation loss: 0.104865 Best loss: 0.104865 Accuracy: 97.97% 4 Validation loss: 0.060471 Best loss: 0.060471 Accuracy: 98.40% 5 Validation loss: 0.069443 Best loss: 0.060471 Accuracy: 98.48% 6 Validation loss: 0.054051 Best loss: 0.054051 Accuracy: 98.40% 7 Validation loss: 0.064788 Best loss: 0.054051 Accuracy: 98.59% 8 Validation loss: 0.064887 Best loss: 0.054051 Accuracy: 98.83% 9 Validation loss: 0.047885 Best loss: 0.047885 Accuracy: 98.79% 10 Validation loss: 0.259117 Best loss: 0.047885 Accuracy: 96.64% 11 Validation loss: 0.073576 Best loss: 0.047885 Accuracy: 98.51% 12 Validation loss: 0.107414 Best loss: 0.047885 Accuracy: 98.32% 13 Validation loss: 0.069720 Best loss: 0.047885 Accuracy: 98.28% 14 Validation loss: 0.054123 Best loss: 0.047885 Accuracy: 98.83% 15 Validation loss: 0.114678 Best loss: 0.047885 Accuracy: 98.91% 16 Validation loss: 0.107343 Best loss: 0.047885 Accuracy: 98.51% 17 Validation loss: 0.096261 Best loss: 0.047885 Accuracy: 98.36% 18 Validation loss: 0.061913 Best loss: 0.047885 Accuracy: 98.63% 19 Validation loss: 0.081590 Best loss: 0.047885 Accuracy: 98.59% 20 Validation loss: 0.074126 Best loss: 0.047885 Accuracy: 98.36% 21 Validation loss: 0.087916 Best loss: 0.047885 Accuracy: 98.75% 22 Validation loss: 0.051912 Best loss: 0.047885 Accuracy: 98.91% 23 Validation loss: 0.050098 Best loss: 0.047885 Accuracy: 98.83% 24 Validation loss: 0.067404 Best loss: 0.047885 Accuracy: 98.87% 25 Validation loss: 0.181788 Best loss: 0.047885 Accuracy: 97.77% 26 Validation loss: 0.104775 Best loss: 0.047885 Accuracy: 97.93% 27 Validation loss: 0.113572 Best loss: 0.047885 Accuracy: 97.89% 28 Validation loss: 0.090700 Best loss: 0.047885 Accuracy: 98.48% 29 Validation loss: 0.104616 Best loss: 0.047885 Accuracy: 98.63% 30 Validation loss: 0.053161 Best loss: 0.047885 Accuracy: 98.87% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>, n_neurons=100, batch_size=100, batch_norm_momentum=0.99, learning_rate=0.1, total= 43.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 0.110524 Best loss: 0.110524 Accuracy: 97.03% 1 Validation loss: 0.092048 Best loss: 0.092048 Accuracy: 97.77% 2 Validation loss: 0.092725 Best loss: 0.092048 Accuracy: 97.81% 3 Validation loss: 0.060941 Best loss: 0.060941 Accuracy: 97.97% 4 Validation loss: 0.063051 Best loss: 0.060941 Accuracy: 98.16% 5 Validation loss: 0.054306 Best loss: 0.054306 Accuracy: 98.63% 6 Validation loss: 0.056113 Best loss: 0.054306 Accuracy: 98.12% 7 Validation loss: 0.062662 Best loss: 0.054306 Accuracy: 98.71% 8 Validation loss: 0.057710 Best loss: 0.054306 Accuracy: 98.71% 9 Validation loss: 0.193188 Best loss: 0.054306 Accuracy: 97.69% 10 Validation loss: 0.092985 Best loss: 0.054306 Accuracy: 98.01% 11 Validation loss: 0.043227 Best loss: 0.043227 Accuracy: 98.83% 12 Validation loss: 0.065010 Best loss: 0.043227 Accuracy: 98.55% 13 Validation loss: 0.071375 Best loss: 0.043227 Accuracy: 98.51% 14 Validation loss: 0.279381 Best loss: 0.043227 Accuracy: 97.77% 15 Validation loss: 0.055243 Best loss: 0.043227 Accuracy: 99.06% 16 Validation loss: 0.053276 Best loss: 0.043227 Accuracy: 98.71% 17 Validation loss: 0.086267 Best loss: 0.043227 Accuracy: 98.40% 18 Validation loss: 0.048165 Best loss: 0.043227 Accuracy: 98.75% 19 Validation loss: 0.072972 Best loss: 0.043227 Accuracy: 98.67% 20 Validation loss: 0.061447 Best loss: 0.043227 Accuracy: 98.71% 21 Validation loss: 0.080065 Best loss: 0.043227 Accuracy: 98.75% 22 Validation loss: 0.210176 Best loss: 0.043227 Accuracy: 98.01% 23 Validation loss: 0.101680 Best loss: 0.043227 Accuracy: 98.63% 24 Validation loss: 0.036191 Best loss: 0.036191 Accuracy: 99.26% 25 Validation loss: 0.042662 Best loss: 0.036191 Accuracy: 99.22% 26 Validation loss: 0.057325 Best loss: 0.036191 Accuracy: 98.94% 27 Validation loss: 0.097505 Best loss: 0.036191 Accuracy: 98.28% 28 Validation loss: 0.160520 Best loss: 0.036191 Accuracy: 98.55% 29 Validation loss: 0.092488 Best loss: 0.036191 Accuracy: 98.98% 30 Validation loss: 0.072455 Best loss: 0.036191 Accuracy: 98.98% 31 Validation loss: 0.072694 Best loss: 0.036191 Accuracy: 99.02% 32 Validation loss: 0.085565 Best loss: 0.036191 Accuracy: 98.59% 33 Validation loss: 0.096016 Best loss: 0.036191 Accuracy: 98.63% 34 Validation loss: 0.093004 Best loss: 0.036191 Accuracy: 98.79% 35 Validation loss: 0.190028 Best loss: 0.036191 Accuracy: 97.97% 36 Validation loss: 0.368073 Best loss: 0.036191 Accuracy: 96.87% 37 Validation loss: 0.252867 Best loss: 0.036191 Accuracy: 98.08% 38 Validation loss: 0.067801 Best loss: 0.036191 Accuracy: 99.10% 39 Validation loss: 0.055809 Best loss: 0.036191 Accuracy: 99.06% 40 Validation loss: 0.089666 Best loss: 0.036191 Accuracy: 98.75% 41 Validation loss: 0.076607 Best loss: 0.036191 Accuracy: 99.06% 42 Validation loss: 0.075137 Best loss: 0.036191 Accuracy: 98.87% 43 Validation loss: 1.604434 Best loss: 0.036191 Accuracy: 96.25% 44 Validation loss: 0.136972 Best loss: 0.036191 Accuracy: 98.12% 45 Validation loss: 0.074747 Best loss: 0.036191 Accuracy: 98.87% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05, total= 1.8min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 0.195579 Best loss: 0.195579 Accuracy: 94.76% 1 Validation loss: 0.079099 Best loss: 0.079099 Accuracy: 97.77% 2 Validation loss: 0.086658 Best loss: 0.079099 Accuracy: 97.73% 3 Validation loss: 0.091559 Best loss: 0.079099 Accuracy: 97.65% 4 Validation loss: 0.081112 Best loss: 0.079099 Accuracy: 97.81% 5 Validation loss: 0.063547 Best loss: 0.063547 Accuracy: 98.44% 6 Validation loss: 0.109898 Best loss: 0.063547 Accuracy: 97.34% 7 Validation loss: 0.080033 Best loss: 0.063547 Accuracy: 97.93% 8 Validation loss: 0.062489 Best loss: 0.062489 Accuracy: 98.63% 9 Validation loss: 0.494636 Best loss: 0.062489 Accuracy: 95.66% 10 Validation loss: 0.147798 Best loss: 0.062489 Accuracy: 97.65% 11 Validation loss: 0.069149 Best loss: 0.062489 Accuracy: 98.75% 12 Validation loss: 0.037797 Best loss: 0.037797 Accuracy: 99.14% 13 Validation loss: 0.045534 Best loss: 0.037797 Accuracy: 98.91% 14 Validation loss: 0.060921 Best loss: 0.037797 Accuracy: 98.44% 15 Validation loss: 0.092521 Best loss: 0.037797 Accuracy: 98.44% 16 Validation loss: 0.179034 Best loss: 0.037797 Accuracy: 97.54% 17 Validation loss: 0.084373 Best loss: 0.037797 Accuracy: 97.73% 18 Validation loss: 0.049661 Best loss: 0.037797 Accuracy: 98.98% 19 Validation loss: 0.035501 Best loss: 0.035501 Accuracy: 99.10% 20 Validation loss: 0.055459 Best loss: 0.035501 Accuracy: 98.87% 21 Validation loss: 0.043159 Best loss: 0.035501 Accuracy: 98.79% 22 Validation loss: 0.063775 Best loss: 0.035501 Accuracy: 98.87% 23 Validation loss: 0.392821 Best loss: 0.035501 Accuracy: 96.29% 24 Validation loss: 0.077653 Best loss: 0.035501 Accuracy: 98.83% 25 Validation loss: 0.042550 Best loss: 0.035501 Accuracy: 98.98% 26 Validation loss: 0.046344 Best loss: 0.035501 Accuracy: 99.22% 27 Validation loss: 0.116444 Best loss: 0.035501 Accuracy: 98.71% 28 Validation loss: 0.053035 Best loss: 0.035501 Accuracy: 98.91% 29 Validation loss: 0.166139 Best loss: 0.035501 Accuracy: 97.89% 30 Validation loss: 0.068722 Best loss: 0.035501 Accuracy: 98.87% 31 Validation loss: 0.075595 Best loss: 0.035501 Accuracy: 98.98% 32 Validation loss: 0.129963 Best loss: 0.035501 Accuracy: 98.59% 33 Validation loss: 0.077116 Best loss: 0.035501 Accuracy: 98.71% 34 Validation loss: 0.066545 Best loss: 0.035501 Accuracy: 98.98% 35 Validation loss: 0.085359 Best loss: 0.035501 Accuracy: 98.79% 36 Validation loss: 0.141340 Best loss: 0.035501 Accuracy: 98.67% 37 Validation loss: 0.112211 Best loss: 0.035501 Accuracy: 98.79% 38 Validation loss: 0.066591 Best loss: 0.035501 Accuracy: 99.10% 39 Validation loss: 0.062816 Best loss: 0.035501 Accuracy: 99.06% 40 Validation loss: 0.775301 Best loss: 0.035501 Accuracy: 97.42% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05, total= 1.6min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05 0 Validation loss: 0.163232 Best loss: 0.163232 Accuracy: 96.01% 1 Validation loss: 0.102235 Best loss: 0.102235 Accuracy: 96.91% 2 Validation loss: 0.077641 Best loss: 0.077641 Accuracy: 97.50% 3 Validation loss: 0.064542 Best loss: 0.064542 Accuracy: 98.12% 4 Validation loss: 0.168254 Best loss: 0.064542 Accuracy: 97.69% 5 Validation loss: 0.055123 Best loss: 0.055123 Accuracy: 98.32% 6 Validation loss: 0.056262 Best loss: 0.055123 Accuracy: 98.48% 7 Validation loss: 0.081612 Best loss: 0.055123 Accuracy: 98.44% 8 Validation loss: 0.076614 Best loss: 0.055123 Accuracy: 98.40% 9 Validation loss: 0.111435 Best loss: 0.055123 Accuracy: 97.50% 10 Validation loss: 0.109507 Best loss: 0.055123 Accuracy: 98.36% 11 Validation loss: 0.056299 Best loss: 0.055123 Accuracy: 98.71% 12 Validation loss: 0.059002 Best loss: 0.055123 Accuracy: 98.40% 13 Validation loss: 0.068690 Best loss: 0.055123 Accuracy: 98.36% 14 Validation loss: 0.043099 Best loss: 0.043099 Accuracy: 98.75% 15 Validation loss: 0.048298 Best loss: 0.043099 Accuracy: 99.14% 16 Validation loss: 0.035734 Best loss: 0.035734 Accuracy: 99.18% 17 Validation loss: 0.072638 Best loss: 0.035734 Accuracy: 98.48% 18 Validation loss: 0.054735 Best loss: 0.035734 Accuracy: 98.67% 19 Validation loss: 0.047238 Best loss: 0.035734 Accuracy: 99.18% 20 Validation loss: 0.076803 Best loss: 0.035734 Accuracy: 98.55% 21 Validation loss: 0.138425 Best loss: 0.035734 Accuracy: 97.77% 22 Validation loss: 0.055587 Best loss: 0.035734 Accuracy: 98.71% 23 Validation loss: 0.046213 Best loss: 0.035734 Accuracy: 99.14% 24 Validation loss: 0.060911 Best loss: 0.035734 Accuracy: 98.83% 25 Validation loss: 0.138837 Best loss: 0.035734 Accuracy: 98.28% 26 Validation loss: 0.133485 Best loss: 0.035734 Accuracy: 98.16% 27 Validation loss: 0.041844 Best loss: 0.035734 Accuracy: 98.83% 28 Validation loss: 0.051681 Best loss: 0.035734 Accuracy: 99.14% 29 Validation loss: 0.066367 Best loss: 0.035734 Accuracy: 98.40% 30 Validation loss: 0.059408 Best loss: 0.035734 Accuracy: 98.98% 31 Validation loss: 0.173327 Best loss: 0.035734 Accuracy: 98.71% 32 Validation loss: 0.104963 Best loss: 0.035734 Accuracy: 98.63% 33 Validation loss: 0.067334 Best loss: 0.035734 Accuracy: 98.87% 34 Validation loss: 0.067494 Best loss: 0.035734 Accuracy: 99.06% 35 Validation loss: 0.053868 Best loss: 0.035734 Accuracy: 99.02% 36 Validation loss: 0.067436 Best loss: 0.035734 Accuracy: 98.94% 37 Validation loss: 0.047526 Best loss: 0.035734 Accuracy: 98.94% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, batch_norm_momentum=0.99, learning_rate=0.05, total= 1.5min
[Parallel(n_jobs=1)]: Done 150 out of 150 | elapsed: 374.0min finished
0 Validation loss: 0.091714 Best loss: 0.091714 Accuracy: 97.69% 1 Validation loss: 0.045306 Best loss: 0.045306 Accuracy: 98.36% 2 Validation loss: 0.036383 Best loss: 0.036383 Accuracy: 98.79% 3 Validation loss: 0.049363 Best loss: 0.036383 Accuracy: 98.63% 4 Validation loss: 0.029773 Best loss: 0.029773 Accuracy: 99.10% 5 Validation loss: 0.029473 Best loss: 0.029473 Accuracy: 99.10% 6 Validation loss: 0.039143 Best loss: 0.029473 Accuracy: 98.87% 7 Validation loss: 0.065406 Best loss: 0.029473 Accuracy: 98.32% 8 Validation loss: 0.046095 Best loss: 0.029473 Accuracy: 98.79% 9 Validation loss: 0.034996 Best loss: 0.029473 Accuracy: 99.06% 10 Validation loss: 0.049377 Best loss: 0.029473 Accuracy: 98.91% 11 Validation loss: 0.039845 Best loss: 0.029473 Accuracy: 99.18% 12 Validation loss: 0.036246 Best loss: 0.029473 Accuracy: 98.98% 13 Validation loss: 0.050732 Best loss: 0.029473 Accuracy: 98.67% 14 Validation loss: 0.027731 Best loss: 0.027731 Accuracy: 99.18% 15 Validation loss: 0.042537 Best loss: 0.027731 Accuracy: 98.79% 16 Validation loss: 0.033484 Best loss: 0.027731 Accuracy: 99.14% 17 Validation loss: 0.030523 Best loss: 0.027731 Accuracy: 99.18% 18 Validation loss: 0.027301 Best loss: 0.027301 Accuracy: 99.37% 19 Validation loss: 0.036797 Best loss: 0.027301 Accuracy: 99.10% 20 Validation loss: 0.034431 Best loss: 0.027301 Accuracy: 99.14% 21 Validation loss: 0.040333 Best loss: 0.027301 Accuracy: 99.14% 22 Validation loss: 0.031700 Best loss: 0.027301 Accuracy: 99.10% 23 Validation loss: 0.043072 Best loss: 0.027301 Accuracy: 99.06% 24 Validation loss: 0.028993 Best loss: 0.027301 Accuracy: 99.18% 25 Validation loss: 0.041151 Best loss: 0.027301 Accuracy: 98.91% 26 Validation loss: 0.029968 Best loss: 0.027301 Accuracy: 99.41% 27 Validation loss: 0.038343 Best loss: 0.027301 Accuracy: 99.22% 28 Validation loss: 0.039325 Best loss: 0.027301 Accuracy: 99.18% 29 Validation loss: 0.037732 Best loss: 0.027301 Accuracy: 99.02% 30 Validation loss: 0.033873 Best loss: 0.027301 Accuracy: 99.14% 31 Validation loss: 0.035938 Best loss: 0.027301 Accuracy: 99.26% 32 Validation loss: 0.036438 Best loss: 0.027301 Accuracy: 99.18% 33 Validation loss: 0.044826 Best loss: 0.027301 Accuracy: 99.14% 34 Validation loss: 0.049675 Best loss: 0.027301 Accuracy: 98.44% 35 Validation loss: 0.040656 Best loss: 0.027301 Accuracy: 99.18% 36 Validation loss: 0.036387 Best loss: 0.027301 Accuracy: 99.26% 37 Validation loss: 0.037122 Best loss: 0.027301 Accuracy: 99.18% 38 Validation loss: 0.031404 Best loss: 0.027301 Accuracy: 99.30% 39 Validation loss: 0.035523 Best loss: 0.027301 Accuracy: 99.37% Early stopping!
Out[130]:
RandomizedSearchCV(cv=None, error_score='raise',
estimator=DNNClassifier(activation=<function elu at 0x7f95d738dbf8>,
batch_norm_momentum=None, batch_size=20, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=100,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42),
fit_params={'y_valid': array([0, 4, ..., 1, 2], dtype=int32), 'X_valid': array([[0., 0., ..., 0., 0.],
[0., 0., ..., 0., 0.],
...,
[0., 0., ..., 0., 0.],
[0., 0., ..., 0., 0.]], dtype=float32), 'n_epochs': 1000},
iid=True, n_iter=50, n_jobs=1,
param_distributions={'activation': [<function relu at 0x7f95d7315f28>, <function elu at 0x7f95d738dbf8>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9400>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c38b9378>], 'n_neurons': [10, 30, 50, 70, 90, 100, 120, 140, 160], 'batch_size': [10, 50, 100, 500], 'batch_norm_momentum': [0.9, 0.95, 0.98, 0.99, 0.999], 'learning_rate': [0.01, 0.02, 0.05, 0.1]},
pre_dispatch='2*n_jobs', random_state=42, refit=True,
return_train_score='warn', scoring=None, verbose=2)
In [131]:
rnd_search_bn.best_params_
Out[131]:
{'activation': <function tensorflow.python.ops.gen_nn_ops.relu(features, name=None)>,
'batch_norm_momentum': 0.99,
'batch_size': 50,
'learning_rate': 0.01,
'n_neurons': 70}
In [132]:
y_pred = rnd_search_bn.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[132]:
0.9937731076084841
Slightly better than earlier: 99.4% vs 99.3%. Let's see if dropout can do better.
8.5.¶
Exercise: is the model overfitting the training set? Try adding dropout to every layer and try again. Does it help?
Let's go back to the best model we trained earlier and see how it performs on the training set:
In [133]:
y_pred = dnn_clf.predict(X_train1)
accuracy_score(y_train1, y_pred)
Out[133]:
0.9967900706184464
The model performs significantly better on the training set than on the test set (99.91% vs 99.32%), which means it is overfitting the training set. A bit of regularization may help. Let's try adding dropout with a 50% dropout rate:
In [134]:
dnn_clf_dropout = DNNClassifier(activation=leaky_relu(alpha=0.1), batch_size=500, learning_rate=0.01,
n_neurons=90, random_state=42,
dropout_rate=0.5)
dnn_clf_dropout.fit(X_train1, y_train1, n_epochs=1000, X_valid=X_valid1, y_valid=y_valid1)
0 Validation loss: 0.118581 Best loss: 0.118581 Accuracy: 97.03% 1 Validation loss: 0.096451 Best loss: 0.096451 Accuracy: 97.26% 2 Validation loss: 0.089992 Best loss: 0.089992 Accuracy: 97.42% 3 Validation loss: 0.078204 Best loss: 0.078204 Accuracy: 97.81% 4 Validation loss: 0.079357 Best loss: 0.078204 Accuracy: 98.08% 5 Validation loss: 0.077832 Best loss: 0.077832 Accuracy: 98.01% 6 Validation loss: 0.080496 Best loss: 0.077832 Accuracy: 97.89% 7 Validation loss: 0.084617 Best loss: 0.077832 Accuracy: 97.73% 8 Validation loss: 0.067436 Best loss: 0.067436 Accuracy: 98.24% 9 Validation loss: 0.076026 Best loss: 0.067436 Accuracy: 98.16% 10 Validation loss: 0.077239 Best loss: 0.067436 Accuracy: 97.81% 11 Validation loss: 0.067822 Best loss: 0.067436 Accuracy: 98.20% 12 Validation loss: 0.068074 Best loss: 0.067436 Accuracy: 98.36% 13 Validation loss: 0.071436 Best loss: 0.067436 Accuracy: 97.97% 14 Validation loss: 0.066191 Best loss: 0.066191 Accuracy: 98.40% 15 Validation loss: 0.068760 Best loss: 0.066191 Accuracy: 98.24% 16 Validation loss: 0.068773 Best loss: 0.066191 Accuracy: 98.36% 17 Validation loss: 0.068815 Best loss: 0.066191 Accuracy: 98.28% 18 Validation loss: 0.071764 Best loss: 0.066191 Accuracy: 98.05% 19 Validation loss: 0.066970 Best loss: 0.066191 Accuracy: 98.16% 20 Validation loss: 0.065300 Best loss: 0.065300 Accuracy: 98.40% 21 Validation loss: 0.065671 Best loss: 0.065300 Accuracy: 98.16% 22 Validation loss: 0.067656 Best loss: 0.065300 Accuracy: 98.28% 23 Validation loss: 0.063430 Best loss: 0.063430 Accuracy: 98.44% 24 Validation loss: 0.076497 Best loss: 0.063430 Accuracy: 98.28% 25 Validation loss: 0.063222 Best loss: 0.063222 Accuracy: 98.20% 26 Validation loss: 0.072677 Best loss: 0.063222 Accuracy: 98.28% 27 Validation loss: 0.086858 Best loss: 0.063222 Accuracy: 98.16% 28 Validation loss: 0.082153 Best loss: 0.063222 Accuracy: 97.97% 29 Validation loss: 0.086823 Best loss: 0.063222 Accuracy: 97.50% 30 Validation loss: 0.117123 Best loss: 0.063222 Accuracy: 97.62% 31 Validation loss: 0.286722 Best loss: 0.063222 Accuracy: 89.09% 32 Validation loss: 0.151667 Best loss: 0.063222 Accuracy: 95.82% 33 Validation loss: 0.159793 Best loss: 0.063222 Accuracy: 95.04% 34 Validation loss: 0.221658 Best loss: 0.063222 Accuracy: 93.67% 35 Validation loss: 0.176935 Best loss: 0.063222 Accuracy: 95.62% 36 Validation loss: 0.134725 Best loss: 0.063222 Accuracy: 95.35% 37 Validation loss: 0.130137 Best loss: 0.063222 Accuracy: 96.72% 38 Validation loss: 0.126741 Best loss: 0.063222 Accuracy: 96.99% 39 Validation loss: 0.121197 Best loss: 0.063222 Accuracy: 97.26% 40 Validation loss: 0.342830 Best loss: 0.063222 Accuracy: 90.11% 41 Validation loss: 0.216394 Best loss: 0.063222 Accuracy: 94.10% 42 Validation loss: 0.196309 Best loss: 0.063222 Accuracy: 93.75% 43 Validation loss: 0.154740 Best loss: 0.063222 Accuracy: 95.70% 44 Validation loss: 0.154868 Best loss: 0.063222 Accuracy: 95.39% 45 Validation loss: 0.152964 Best loss: 0.063222 Accuracy: 95.15% 46 Validation loss: 0.142038 Best loss: 0.063222 Accuracy: 95.82% Early stopping!
Out[134]:
DNNClassifier(activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809f28>,
batch_norm_momentum=None, batch_size=500, dropout_rate=0.5,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=90,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42)
The best params are reached during epoch 23. Dropout somewhat slowed down convergence.
Let's check the accuracy:
In [135]:
y_pred = dnn_clf_dropout.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[135]:
0.9856003113446196
We are out of luck, dropout does not seem to help either. Let's try tuning the hyperparameters, perhaps we can squeeze a bit more performance out of this model:
In [136]:
from sklearn.model_selection import RandomizedSearchCV
param_distribs = {
"n_neurons": [10, 30, 50, 70, 90, 100, 120, 140, 160],
"batch_size": [10, 50, 100, 500],
"learning_rate": [0.01, 0.02, 0.05, 0.1],
"activation": [tf.nn.relu, tf.nn.elu, leaky_relu(alpha=0.01), leaky_relu(alpha=0.1)],
# you could also try exploring different numbers of hidden layers, different optimizers, etc.
#"n_hidden_layers": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
#"optimizer_class": [tf.train.AdamOptimizer, partial(tf.train.MomentumOptimizer, momentum=0.95)],
"dropout_rate": [0.2, 0.3, 0.4, 0.5, 0.6],
}
rnd_search_dropout = RandomizedSearchCV(DNNClassifier(random_state=42), param_distribs, n_iter=50,
fit_params={"X_valid": X_valid1, "y_valid": y_valid1, "n_epochs": 1000},
random_state=42, verbose=2)
rnd_search_dropout.fit(X_train1, y_train1)
Fitting 3 folds for each of 50 candidates, totalling 150 fits [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5
/home/ageron/.virtualenvs/ml/lib/python3.5/site-packages/sklearn/model_selection/_search.py:584: DeprecationWarning: "fit_params" as a constructor argument was deprecated in version 0.19 and will be removed in version 0.21. Pass fit parameters to the "fit" method instead. '"fit" method instead.', DeprecationWarning)
0 Validation loss: 0.208147 Best loss: 0.208147 Accuracy: 94.06% 1 Validation loss: 0.169655 Best loss: 0.169655 Accuracy: 95.39% 2 Validation loss: 0.209455 Best loss: 0.169655 Accuracy: 95.66% 3 Validation loss: 0.186685 Best loss: 0.169655 Accuracy: 94.76% 4 Validation loss: 0.190829 Best loss: 0.169655 Accuracy: 95.97% 5 Validation loss: 0.190536 Best loss: 0.169655 Accuracy: 95.70% 6 Validation loss: 0.180128 Best loss: 0.169655 Accuracy: 95.39% 7 Validation loss: 0.236844 Best loss: 0.169655 Accuracy: 96.48% 8 Validation loss: 0.170223 Best loss: 0.169655 Accuracy: 95.90% 9 Validation loss: 0.206953 Best loss: 0.169655 Accuracy: 95.58% 10 Validation loss: 0.180888 Best loss: 0.169655 Accuracy: 95.90% 11 Validation loss: 0.167653 Best loss: 0.167653 Accuracy: 95.90% 12 Validation loss: 0.169967 Best loss: 0.167653 Accuracy: 95.97% 13 Validation loss: 0.267441 Best loss: 0.167653 Accuracy: 95.93% 14 Validation loss: 0.158053 Best loss: 0.158053 Accuracy: 96.36% 15 Validation loss: 0.228234 Best loss: 0.158053 Accuracy: 94.18% 16 Validation loss: 0.170770 Best loss: 0.158053 Accuracy: 96.13% 17 Validation loss: 0.181792 Best loss: 0.158053 Accuracy: 94.92% 18 Validation loss: 0.199189 Best loss: 0.158053 Accuracy: 95.62% 19 Validation loss: 0.153413 Best loss: 0.153413 Accuracy: 96.05% 20 Validation loss: 0.175345 Best loss: 0.153413 Accuracy: 95.39% 21 Validation loss: 0.152513 Best loss: 0.152513 Accuracy: 95.86% 22 Validation loss: 0.151548 Best loss: 0.151548 Accuracy: 96.52% 23 Validation loss: 0.173999 Best loss: 0.151548 Accuracy: 95.97% 24 Validation loss: 0.182452 Best loss: 0.151548 Accuracy: 96.09% 25 Validation loss: 0.165147 Best loss: 0.151548 Accuracy: 96.44% 26 Validation loss: 0.166372 Best loss: 0.151548 Accuracy: 96.52% 27 Validation loss: 0.168875 Best loss: 0.151548 Accuracy: 96.33% 28 Validation loss: 0.183164 Best loss: 0.151548 Accuracy: 95.86% 29 Validation loss: 0.169824 Best loss: 0.151548 Accuracy: 96.13% 30 Validation loss: 0.162581 Best loss: 0.151548 Accuracy: 96.17% 31 Validation loss: 0.173767 Best loss: 0.151548 Accuracy: 96.05% 32 Validation loss: 0.202724 Best loss: 0.151548 Accuracy: 96.29% 33 Validation loss: 0.152097 Best loss: 0.151548 Accuracy: 96.36% 34 Validation loss: 0.152260 Best loss: 0.151548 Accuracy: 96.76% 35 Validation loss: 0.153306 Best loss: 0.151548 Accuracy: 96.33% 36 Validation loss: 0.131524 Best loss: 0.131524 Accuracy: 96.60% 37 Validation loss: 0.155232 Best loss: 0.131524 Accuracy: 96.76% 38 Validation loss: 0.148815 Best loss: 0.131524 Accuracy: 96.87% 39 Validation loss: 0.148476 Best loss: 0.131524 Accuracy: 96.56% 40 Validation loss: 0.165940 Best loss: 0.131524 Accuracy: 96.64% 41 Validation loss: 0.147053 Best loss: 0.131524 Accuracy: 96.87% 42 Validation loss: 0.145738 Best loss: 0.131524 Accuracy: 97.07% 43 Validation loss: 0.189651 Best loss: 0.131524 Accuracy: 95.93% 44 Validation loss: 0.159221 Best loss: 0.131524 Accuracy: 96.79% 45 Validation loss: 0.183259 Best loss: 0.131524 Accuracy: 96.48% 46 Validation loss: 0.169637 Best loss: 0.131524 Accuracy: 96.21% 47 Validation loss: 0.287415 Best loss: 0.131524 Accuracy: 96.76% 48 Validation loss: 0.158538 Best loss: 0.131524 Accuracy: 96.64% 49 Validation loss: 0.186645 Best loss: 0.131524 Accuracy: 96.64% 50 Validation loss: 0.158634 Best loss: 0.131524 Accuracy: 96.91% 51 Validation loss: 0.186893 Best loss: 0.131524 Accuracy: 96.68% 52 Validation loss: 0.165234 Best loss: 0.131524 Accuracy: 96.95% 53 Validation loss: 0.150569 Best loss: 0.131524 Accuracy: 96.64% 54 Validation loss: 0.134656 Best loss: 0.131524 Accuracy: 96.99% 55 Validation loss: 0.150487 Best loss: 0.131524 Accuracy: 97.11% 56 Validation loss: 0.141543 Best loss: 0.131524 Accuracy: 96.64% 57 Validation loss: 0.144569 Best loss: 0.131524 Accuracy: 96.76% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 33.4s
[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 33.5s remaining: 0.0s
[CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.240026 Best loss: 0.240026 Accuracy: 93.59% 1 Validation loss: 0.207125 Best loss: 0.207125 Accuracy: 94.76% 2 Validation loss: 0.166744 Best loss: 0.166744 Accuracy: 95.39% 3 Validation loss: 0.167589 Best loss: 0.166744 Accuracy: 95.78% 4 Validation loss: 0.155207 Best loss: 0.155207 Accuracy: 95.90% 5 Validation loss: 0.187034 Best loss: 0.155207 Accuracy: 94.45% 6 Validation loss: 0.198623 Best loss: 0.155207 Accuracy: 95.15% 7 Validation loss: 0.166998 Best loss: 0.155207 Accuracy: 95.74% 8 Validation loss: 0.149803 Best loss: 0.149803 Accuracy: 96.76% 9 Validation loss: 0.153711 Best loss: 0.149803 Accuracy: 96.09% 10 Validation loss: 0.154203 Best loss: 0.149803 Accuracy: 96.40% 11 Validation loss: 0.138522 Best loss: 0.138522 Accuracy: 96.68% 12 Validation loss: 0.182553 Best loss: 0.138522 Accuracy: 96.52% 13 Validation loss: 0.175780 Best loss: 0.138522 Accuracy: 96.64% 14 Validation loss: 0.183933 Best loss: 0.138522 Accuracy: 95.90% 15 Validation loss: 0.174764 Best loss: 0.138522 Accuracy: 95.93% 16 Validation loss: 0.190883 Best loss: 0.138522 Accuracy: 95.66% 17 Validation loss: 0.155555 Best loss: 0.138522 Accuracy: 96.40% 18 Validation loss: 0.188842 Best loss: 0.138522 Accuracy: 96.56% 19 Validation loss: 0.159297 Best loss: 0.138522 Accuracy: 96.44% 20 Validation loss: 0.160345 Best loss: 0.138522 Accuracy: 96.64% 21 Validation loss: 0.158270 Best loss: 0.138522 Accuracy: 95.97% 22 Validation loss: 0.140859 Best loss: 0.138522 Accuracy: 96.87% 23 Validation loss: 0.146913 Best loss: 0.138522 Accuracy: 96.83% 24 Validation loss: 0.167238 Best loss: 0.138522 Accuracy: 96.87% 25 Validation loss: 0.172726 Best loss: 0.138522 Accuracy: 95.86% 26 Validation loss: 0.182239 Best loss: 0.138522 Accuracy: 95.97% 27 Validation loss: 0.146947 Best loss: 0.138522 Accuracy: 96.72% 28 Validation loss: 0.144365 Best loss: 0.138522 Accuracy: 96.91% 29 Validation loss: 0.148439 Best loss: 0.138522 Accuracy: 96.87% 30 Validation loss: 0.184532 Best loss: 0.138522 Accuracy: 95.62% 31 Validation loss: 0.187510 Best loss: 0.138522 Accuracy: 96.13% 32 Validation loss: 0.171149 Best loss: 0.138522 Accuracy: 95.78% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 19.5s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.206202 Best loss: 0.206202 Accuracy: 94.76% 1 Validation loss: 0.228780 Best loss: 0.206202 Accuracy: 94.80% 2 Validation loss: 0.168821 Best loss: 0.168821 Accuracy: 95.58% 3 Validation loss: 0.191993 Best loss: 0.168821 Accuracy: 95.93% 4 Validation loss: 0.170500 Best loss: 0.168821 Accuracy: 95.97% 5 Validation loss: 0.163036 Best loss: 0.163036 Accuracy: 96.33% 6 Validation loss: 0.172709 Best loss: 0.163036 Accuracy: 95.90% 7 Validation loss: 0.153090 Best loss: 0.153090 Accuracy: 96.52% 8 Validation loss: 0.142134 Best loss: 0.142134 Accuracy: 96.52% 9 Validation loss: 0.174924 Best loss: 0.142134 Accuracy: 96.44% 10 Validation loss: 0.163793 Best loss: 0.142134 Accuracy: 96.36% 11 Validation loss: 0.156676 Best loss: 0.142134 Accuracy: 95.93% 12 Validation loss: 0.134547 Best loss: 0.134547 Accuracy: 96.60% 13 Validation loss: 0.140250 Best loss: 0.134547 Accuracy: 96.99% 14 Validation loss: 0.150798 Best loss: 0.134547 Accuracy: 96.72% 15 Validation loss: 0.162616 Best loss: 0.134547 Accuracy: 96.29% 16 Validation loss: 0.146620 Best loss: 0.134547 Accuracy: 96.91% 17 Validation loss: 0.153465 Best loss: 0.134547 Accuracy: 96.68% 18 Validation loss: 0.138830 Best loss: 0.134547 Accuracy: 97.03% 19 Validation loss: 0.138804 Best loss: 0.134547 Accuracy: 97.30% 20 Validation loss: 0.132555 Best loss: 0.132555 Accuracy: 97.11% 21 Validation loss: 0.133510 Best loss: 0.132555 Accuracy: 97.07% 22 Validation loss: 0.154822 Best loss: 0.132555 Accuracy: 96.52% 23 Validation loss: 0.130438 Best loss: 0.130438 Accuracy: 97.42% 24 Validation loss: 0.151102 Best loss: 0.130438 Accuracy: 96.76% 25 Validation loss: 0.155843 Best loss: 0.130438 Accuracy: 96.76% 26 Validation loss: 0.153278 Best loss: 0.130438 Accuracy: 96.33% 27 Validation loss: 0.152885 Best loss: 0.130438 Accuracy: 97.07% 28 Validation loss: 0.157696 Best loss: 0.130438 Accuracy: 96.68% 29 Validation loss: 0.147186 Best loss: 0.130438 Accuracy: 97.07% 30 Validation loss: 0.160445 Best loss: 0.130438 Accuracy: 97.11% 31 Validation loss: 0.150231 Best loss: 0.130438 Accuracy: 96.68% 32 Validation loss: 0.145216 Best loss: 0.130438 Accuracy: 96.95% 33 Validation loss: 0.136285 Best loss: 0.130438 Accuracy: 97.19% 34 Validation loss: 0.134166 Best loss: 0.130438 Accuracy: 96.95% 35 Validation loss: 0.142107 Best loss: 0.130438 Accuracy: 97.07% 36 Validation loss: 0.159939 Best loss: 0.130438 Accuracy: 97.07% 37 Validation loss: 0.151128 Best loss: 0.130438 Accuracy: 97.11% 38 Validation loss: 0.139211 Best loss: 0.130438 Accuracy: 97.07% 39 Validation loss: 0.144337 Best loss: 0.130438 Accuracy: 97.46% 40 Validation loss: 0.145743 Best loss: 0.130438 Accuracy: 97.34% 41 Validation loss: 0.156749 Best loss: 0.130438 Accuracy: 96.36% 42 Validation loss: 0.145559 Best loss: 0.130438 Accuracy: 96.99% 43 Validation loss: 0.151865 Best loss: 0.130438 Accuracy: 97.50% 44 Validation loss: 0.175898 Best loss: 0.130438 Accuracy: 97.34% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 26.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 540.385437 Best loss: 540.385437 Accuracy: 19.82% 1 Validation loss: 4.638870 Best loss: 4.638870 Accuracy: 42.34% 2 Validation loss: 1.993660 Best loss: 1.993660 Accuracy: 38.74% 3 Validation loss: 1.470787 Best loss: 1.470787 Accuracy: 41.83% 4 Validation loss: 5.944736 Best loss: 1.470787 Accuracy: 24.00% 5 Validation loss: 3.842204 Best loss: 1.470787 Accuracy: 22.52% 6 Validation loss: 2.563682 Best loss: 1.470787 Accuracy: 33.89% 7 Validation loss: 28.500584 Best loss: 1.470787 Accuracy: 37.84% 8 Validation loss: 26.858074 Best loss: 1.470787 Accuracy: 32.60% 9 Validation loss: 43.052769 Best loss: 1.470787 Accuracy: 20.13% 10 Validation loss: 78.856964 Best loss: 1.470787 Accuracy: 21.07% 11 Validation loss: 65.664299 Best loss: 1.470787 Accuracy: 39.17% 12 Validation loss: 14.371526 Best loss: 1.470787 Accuracy: 39.21% 13 Validation loss: 21.005749 Best loss: 1.470787 Accuracy: 42.18% 14 Validation loss: 21.924700 Best loss: 1.470787 Accuracy: 24.00% 15 Validation loss: 10.286045 Best loss: 1.470787 Accuracy: 36.86% 16 Validation loss: 9.831418 Best loss: 1.470787 Accuracy: 39.87% 17 Validation loss: 4.587852 Best loss: 1.470787 Accuracy: 43.24% 18 Validation loss: 4.308688 Best loss: 1.470787 Accuracy: 40.07% 19 Validation loss: 11.858151 Best loss: 1.470787 Accuracy: 34.17% 20 Validation loss: 38.555813 Best loss: 1.470787 Accuracy: 25.84% 21 Validation loss: 10950.884766 Best loss: 1.470787 Accuracy: 32.29% 22 Validation loss: 67.709435 Best loss: 1.470787 Accuracy: 21.38% 23 Validation loss: 1316.163452 Best loss: 1.470787 Accuracy: 23.81% 24 Validation loss: 68.765282 Best loss: 1.470787 Accuracy: 38.12% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2, total= 2.5min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 18.608973 Best loss: 18.608973 Accuracy: 19.27% 1 Validation loss: 5.048411 Best loss: 5.048411 Accuracy: 23.73% 2 Validation loss: 210.578995 Best loss: 5.048411 Accuracy: 19.08% 3 Validation loss: 8.198708 Best loss: 5.048411 Accuracy: 31.16% 4 Validation loss: 123.619377 Best loss: 5.048411 Accuracy: 19.27% 5 Validation loss: 31.420116 Best loss: 5.048411 Accuracy: 26.90% 6 Validation loss: 36.633755 Best loss: 5.048411 Accuracy: 19.98% 7 Validation loss: 15.920717 Best loss: 5.048411 Accuracy: 21.38% 8 Validation loss: 9.562019 Best loss: 5.048411 Accuracy: 30.69% 9 Validation loss: 10.238075 Best loss: 5.048411 Accuracy: 26.62% 10 Validation loss: 39.194267 Best loss: 5.048411 Accuracy: 20.80% 11 Validation loss: 96.090332 Best loss: 5.048411 Accuracy: 21.77% 12 Validation loss: 67.080772 Best loss: 5.048411 Accuracy: 36.32% 13 Validation loss: 86.238029 Best loss: 5.048411 Accuracy: 19.66% 14 Validation loss: 33.135883 Best loss: 5.048411 Accuracy: 40.66% 15 Validation loss: 31.783459 Best loss: 5.048411 Accuracy: 33.42% 16 Validation loss: 33.521206 Best loss: 5.048411 Accuracy: 36.86% 17 Validation loss: 25.188196 Best loss: 5.048411 Accuracy: 47.07% 18 Validation loss: 10.756371 Best loss: 5.048411 Accuracy: 44.41% 19 Validation loss: 7.573103 Best loss: 5.048411 Accuracy: 57.31% 20 Validation loss: 43.942066 Best loss: 5.048411 Accuracy: 35.30% 21 Validation loss: 33.476387 Best loss: 5.048411 Accuracy: 35.97% 22 Validation loss: 61.821606 Best loss: 5.048411 Accuracy: 22.44% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2, total= 2.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 2.253963 Best loss: 2.253963 Accuracy: 27.95% 1 Validation loss: 55.912537 Best loss: 2.253963 Accuracy: 27.05% 2 Validation loss: 3.027785 Best loss: 2.253963 Accuracy: 36.08% 3 Validation loss: 2.312928 Best loss: 2.253963 Accuracy: 41.13% 4 Validation loss: 28.319826 Best loss: 2.253963 Accuracy: 18.88% 5 Validation loss: 13.153781 Best loss: 2.253963 Accuracy: 27.64% 6 Validation loss: 4.428048 Best loss: 2.253963 Accuracy: 29.52% 7 Validation loss: 55.280334 Best loss: 2.253963 Accuracy: 19.08% 8 Validation loss: 47.135757 Best loss: 2.253963 Accuracy: 18.73% 9 Validation loss: 31.454323 Best loss: 2.253963 Accuracy: 22.79% 10 Validation loss: 12.263543 Best loss: 2.253963 Accuracy: 28.85% 11 Validation loss: 17.151031 Best loss: 2.253963 Accuracy: 23.42% 12 Validation loss: 20.345594 Best loss: 2.253963 Accuracy: 29.01% 13 Validation loss: 10.082931 Best loss: 2.253963 Accuracy: 32.17% 14 Validation loss: 7.329944 Best loss: 2.253963 Accuracy: 38.90% 15 Validation loss: 279.085144 Best loss: 2.253963 Accuracy: 10.71% 16 Validation loss: 55.837307 Best loss: 2.253963 Accuracy: 21.62% 17 Validation loss: 106.847939 Best loss: 2.253963 Accuracy: 23.34% 18 Validation loss: 203.836548 Best loss: 2.253963 Accuracy: 35.93% 19 Validation loss: 68.018837 Best loss: 2.253963 Accuracy: 24.78% 20 Validation loss: 55.064564 Best loss: 2.253963 Accuracy: 20.91% 21 Validation loss: 88.073547 Best loss: 2.253963 Accuracy: 36.83% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=10, learning_rate=0.02, dropout_rate=0.2, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3 0 Validation loss: 0.138104 Best loss: 0.138104 Accuracy: 96.44% 1 Validation loss: 0.102204 Best loss: 0.102204 Accuracy: 97.34% 2 Validation loss: 0.115205 Best loss: 0.102204 Accuracy: 96.52% 3 Validation loss: 0.124016 Best loss: 0.102204 Accuracy: 96.87% 4 Validation loss: 0.103200 Best loss: 0.102204 Accuracy: 97.11% 5 Validation loss: 0.108648 Best loss: 0.102204 Accuracy: 97.30% 6 Validation loss: 0.106446 Best loss: 0.102204 Accuracy: 96.99% 7 Validation loss: 0.095047 Best loss: 0.095047 Accuracy: 97.50% 8 Validation loss: 0.089223 Best loss: 0.089223 Accuracy: 97.54% 9 Validation loss: 0.105630 Best loss: 0.089223 Accuracy: 97.85% 10 Validation loss: 0.101586 Best loss: 0.089223 Accuracy: 97.93% 11 Validation loss: 0.098749 Best loss: 0.089223 Accuracy: 97.62% 12 Validation loss: 0.100259 Best loss: 0.089223 Accuracy: 97.11% 13 Validation loss: 0.098689 Best loss: 0.089223 Accuracy: 97.62% 14 Validation loss: 0.105920 Best loss: 0.089223 Accuracy: 97.34% 15 Validation loss: 0.111270 Best loss: 0.089223 Accuracy: 97.26% 16 Validation loss: 0.145617 Best loss: 0.089223 Accuracy: 97.26% 17 Validation loss: 0.106696 Best loss: 0.089223 Accuracy: 97.26% 18 Validation loss: 0.098962 Best loss: 0.089223 Accuracy: 97.65% 19 Validation loss: 0.100634 Best loss: 0.089223 Accuracy: 97.58% 20 Validation loss: 0.095513 Best loss: 0.089223 Accuracy: 97.54% 21 Validation loss: 0.096121 Best loss: 0.089223 Accuracy: 98.12% 22 Validation loss: 0.085915 Best loss: 0.085915 Accuracy: 98.01% 23 Validation loss: 0.101000 Best loss: 0.085915 Accuracy: 97.77% 24 Validation loss: 0.101097 Best loss: 0.085915 Accuracy: 97.85% 25 Validation loss: 0.090689 Best loss: 0.085915 Accuracy: 97.65% 26 Validation loss: 0.116989 Best loss: 0.085915 Accuracy: 97.62% 27 Validation loss: 0.100028 Best loss: 0.085915 Accuracy: 97.73% 28 Validation loss: 0.082633 Best loss: 0.082633 Accuracy: 98.05% 29 Validation loss: 0.112337 Best loss: 0.082633 Accuracy: 97.38% 30 Validation loss: 0.099720 Best loss: 0.082633 Accuracy: 97.42% 31 Validation loss: 0.114167 Best loss: 0.082633 Accuracy: 97.19% 32 Validation loss: 0.098532 Best loss: 0.082633 Accuracy: 97.42% 33 Validation loss: 0.083535 Best loss: 0.082633 Accuracy: 97.81% 34 Validation loss: 0.081377 Best loss: 0.081377 Accuracy: 98.05% 35 Validation loss: 0.094226 Best loss: 0.081377 Accuracy: 97.89% 36 Validation loss: 0.113220 Best loss: 0.081377 Accuracy: 97.65% 37 Validation loss: 0.097129 Best loss: 0.081377 Accuracy: 97.81% 38 Validation loss: 0.092850 Best loss: 0.081377 Accuracy: 98.05% 39 Validation loss: 0.090189 Best loss: 0.081377 Accuracy: 98.20% 40 Validation loss: 0.090994 Best loss: 0.081377 Accuracy: 97.93% 41 Validation loss: 0.088680 Best loss: 0.081377 Accuracy: 98.01% 42 Validation loss: 0.138542 Best loss: 0.081377 Accuracy: 97.42% 43 Validation loss: 0.195438 Best loss: 0.081377 Accuracy: 97.15% 44 Validation loss: 0.132391 Best loss: 0.081377 Accuracy: 97.38% 45 Validation loss: 0.116481 Best loss: 0.081377 Accuracy: 97.69% 46 Validation loss: 0.109455 Best loss: 0.081377 Accuracy: 97.58% 47 Validation loss: 0.091082 Best loss: 0.081377 Accuracy: 98.05% 48 Validation loss: 0.089890 Best loss: 0.081377 Accuracy: 97.81% 49 Validation loss: 0.094667 Best loss: 0.081377 Accuracy: 98.05% 50 Validation loss: 0.105719 Best loss: 0.081377 Accuracy: 97.65% 51 Validation loss: 0.124267 Best loss: 0.081377 Accuracy: 97.77% 52 Validation loss: 0.112946 Best loss: 0.081377 Accuracy: 97.85% 53 Validation loss: 0.100934 Best loss: 0.081377 Accuracy: 98.08% 54 Validation loss: 0.107642 Best loss: 0.081377 Accuracy: 97.81% 55 Validation loss: 0.100630 Best loss: 0.081377 Accuracy: 97.89% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3, total= 1.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3 0 Validation loss: 0.131115 Best loss: 0.131115 Accuracy: 96.52% 1 Validation loss: 0.111269 Best loss: 0.111269 Accuracy: 97.03% 2 Validation loss: 0.092469 Best loss: 0.092469 Accuracy: 97.42% 3 Validation loss: 0.098594 Best loss: 0.092469 Accuracy: 97.38% 4 Validation loss: 0.087935 Best loss: 0.087935 Accuracy: 97.69% 5 Validation loss: 0.099145 Best loss: 0.087935 Accuracy: 97.26% 6 Validation loss: 0.088020 Best loss: 0.087935 Accuracy: 97.81% 7 Validation loss: 0.094220 Best loss: 0.087935 Accuracy: 97.89% 8 Validation loss: 0.102825 Best loss: 0.087935 Accuracy: 97.62% 9 Validation loss: 0.110335 Best loss: 0.087935 Accuracy: 97.11% 10 Validation loss: 0.094501 Best loss: 0.087935 Accuracy: 97.81% 11 Validation loss: 0.097417 Best loss: 0.087935 Accuracy: 97.38% 12 Validation loss: 0.081971 Best loss: 0.081971 Accuracy: 97.65% 13 Validation loss: 0.084068 Best loss: 0.081971 Accuracy: 97.97% 14 Validation loss: 0.093110 Best loss: 0.081971 Accuracy: 97.93% 15 Validation loss: 0.089052 Best loss: 0.081971 Accuracy: 97.93% 16 Validation loss: 0.101572 Best loss: 0.081971 Accuracy: 97.11% 17 Validation loss: 0.093324 Best loss: 0.081971 Accuracy: 97.58% 18 Validation loss: 0.085846 Best loss: 0.081971 Accuracy: 97.97% 19 Validation loss: 0.107402 Best loss: 0.081971 Accuracy: 97.69% 20 Validation loss: 0.122688 Best loss: 0.081971 Accuracy: 96.76% 21 Validation loss: 0.098650 Best loss: 0.081971 Accuracy: 97.73% 22 Validation loss: 0.086271 Best loss: 0.081971 Accuracy: 97.77% 23 Validation loss: 0.094498 Best loss: 0.081971 Accuracy: 97.89% 24 Validation loss: 0.095181 Best loss: 0.081971 Accuracy: 97.85% 25 Validation loss: 0.101485 Best loss: 0.081971 Accuracy: 97.77% 26 Validation loss: 0.097053 Best loss: 0.081971 Accuracy: 97.81% 27 Validation loss: 0.084810 Best loss: 0.081971 Accuracy: 98.01% 28 Validation loss: 0.087852 Best loss: 0.081971 Accuracy: 97.62% 29 Validation loss: 0.096378 Best loss: 0.081971 Accuracy: 97.93% 30 Validation loss: 0.091144 Best loss: 0.081971 Accuracy: 97.97% 31 Validation loss: 0.091162 Best loss: 0.081971 Accuracy: 98.20% 32 Validation loss: 0.160510 Best loss: 0.081971 Accuracy: 96.76% 33 Validation loss: 0.140137 Best loss: 0.081971 Accuracy: 97.50% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3, total= 43.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3 0 Validation loss: 0.131255 Best loss: 0.131255 Accuracy: 96.05% 1 Validation loss: 0.114497 Best loss: 0.114497 Accuracy: 96.68% 2 Validation loss: 0.103010 Best loss: 0.103010 Accuracy: 97.26% 3 Validation loss: 0.091421 Best loss: 0.091421 Accuracy: 97.89% 4 Validation loss: 0.095207 Best loss: 0.091421 Accuracy: 97.42% 5 Validation loss: 0.098169 Best loss: 0.091421 Accuracy: 97.26% 6 Validation loss: 0.084248 Best loss: 0.084248 Accuracy: 97.54% 7 Validation loss: 0.089561 Best loss: 0.084248 Accuracy: 97.73% 8 Validation loss: 0.094125 Best loss: 0.084248 Accuracy: 97.38% 9 Validation loss: 0.087678 Best loss: 0.084248 Accuracy: 97.93% 10 Validation loss: 0.083470 Best loss: 0.083470 Accuracy: 97.77% 11 Validation loss: 0.145871 Best loss: 0.083470 Accuracy: 97.81% 12 Validation loss: 0.099017 Best loss: 0.083470 Accuracy: 97.73% 13 Validation loss: 0.087035 Best loss: 0.083470 Accuracy: 97.77% 14 Validation loss: 0.107481 Best loss: 0.083470 Accuracy: 97.62% 15 Validation loss: 0.098975 Best loss: 0.083470 Accuracy: 97.73% 16 Validation loss: 0.093997 Best loss: 0.083470 Accuracy: 97.50% 17 Validation loss: 0.087387 Best loss: 0.083470 Accuracy: 97.85% 18 Validation loss: 0.102026 Best loss: 0.083470 Accuracy: 97.81% 19 Validation loss: 0.112889 Best loss: 0.083470 Accuracy: 97.42% 20 Validation loss: 0.104606 Best loss: 0.083470 Accuracy: 97.81% 21 Validation loss: 0.091629 Best loss: 0.083470 Accuracy: 97.81% 22 Validation loss: 0.095582 Best loss: 0.083470 Accuracy: 97.50% 23 Validation loss: 0.082873 Best loss: 0.082873 Accuracy: 97.73% 24 Validation loss: 0.081464 Best loss: 0.081464 Accuracy: 98.05% 25 Validation loss: 0.081002 Best loss: 0.081002 Accuracy: 97.81% 26 Validation loss: 0.074711 Best loss: 0.074711 Accuracy: 98.05% 27 Validation loss: 0.075872 Best loss: 0.074711 Accuracy: 98.12% 28 Validation loss: 0.079519 Best loss: 0.074711 Accuracy: 97.73% 29 Validation loss: 0.088652 Best loss: 0.074711 Accuracy: 97.93% 30 Validation loss: 0.107547 Best loss: 0.074711 Accuracy: 97.54% 31 Validation loss: 0.097999 Best loss: 0.074711 Accuracy: 97.54% 32 Validation loss: 0.085025 Best loss: 0.074711 Accuracy: 97.89% 33 Validation loss: 0.084994 Best loss: 0.074711 Accuracy: 98.32% 34 Validation loss: 0.101350 Best loss: 0.074711 Accuracy: 97.58% 35 Validation loss: 0.108020 Best loss: 0.074711 Accuracy: 97.89% 36 Validation loss: 0.085137 Best loss: 0.074711 Accuracy: 97.85% 37 Validation loss: 0.090264 Best loss: 0.074711 Accuracy: 98.16% 38 Validation loss: 0.090736 Best loss: 0.074711 Accuracy: 98.05% 39 Validation loss: 0.085842 Best loss: 0.074711 Accuracy: 98.24% 40 Validation loss: 0.090555 Best loss: 0.074711 Accuracy: 97.93% 41 Validation loss: 0.110908 Best loss: 0.074711 Accuracy: 97.73% 42 Validation loss: 0.095620 Best loss: 0.074711 Accuracy: 98.05% 43 Validation loss: 0.098040 Best loss: 0.074711 Accuracy: 98.32% 44 Validation loss: 0.088700 Best loss: 0.074711 Accuracy: 98.12% 45 Validation loss: 0.080355 Best loss: 0.074711 Accuracy: 98.01% 46 Validation loss: 0.110747 Best loss: 0.074711 Accuracy: 97.81% 47 Validation loss: 0.112923 Best loss: 0.074711 Accuracy: 97.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=30, batch_size=50, learning_rate=0.01, dropout_rate=0.3, total= 1.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 57.924992 Best loss: 57.924992 Accuracy: 18.73% 1 Validation loss: 67.971397 Best loss: 57.924992 Accuracy: 19.12% 2 Validation loss: 287.650024 Best loss: 57.924992 Accuracy: 19.27% 3 Validation loss: 504.417267 Best loss: 57.924992 Accuracy: 19.27% 4 Validation loss: 1386.053955 Best loss: 57.924992 Accuracy: 19.08% 5 Validation loss: 1012.425842 Best loss: 57.924992 Accuracy: 22.01% 6 Validation loss: 2088.165771 Best loss: 57.924992 Accuracy: 19.08% 7 Validation loss: 3131.295654 Best loss: 57.924992 Accuracy: 19.27% 8 Validation loss: 1843.685669 Best loss: 57.924992 Accuracy: 18.73% 9 Validation loss: 1221.986450 Best loss: 57.924992 Accuracy: 19.27% 10 Validation loss: 1141.766846 Best loss: 57.924992 Accuracy: 19.08% 11 Validation loss: 2916.219238 Best loss: 57.924992 Accuracy: 18.73% 12 Validation loss: 4000.690430 Best loss: 57.924992 Accuracy: 18.73% 13 Validation loss: 3130.796631 Best loss: 57.924992 Accuracy: 22.60% 14 Validation loss: 12853.981445 Best loss: 57.924992 Accuracy: 19.08% 15 Validation loss: 11819.839844 Best loss: 57.924992 Accuracy: 19.27% 16 Validation loss: 6149.569824 Best loss: 57.924992 Accuracy: 18.73% 17 Validation loss: 2965.763184 Best loss: 57.924992 Accuracy: 28.62% 18 Validation loss: 7543.196289 Best loss: 57.924992 Accuracy: 20.91% 19 Validation loss: 10056.364258 Best loss: 57.924992 Accuracy: 19.27% 20 Validation loss: 3027.124268 Best loss: 57.924992 Accuracy: 23.18% 21 Validation loss: 4642.802734 Best loss: 57.924992 Accuracy: 22.01% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 1.509603 Best loss: 1.509603 Accuracy: 36.63% 1 Validation loss: 64.383682 Best loss: 1.509603 Accuracy: 18.73% 2 Validation loss: 435.984497 Best loss: 1.509603 Accuracy: 19.27% 3 Validation loss: 967.650513 Best loss: 1.509603 Accuracy: 19.27% 4 Validation loss: 1632.163208 Best loss: 1.509603 Accuracy: 19.08% 5 Validation loss: 477.122314 Best loss: 1.509603 Accuracy: 20.91% 6 Validation loss: 2680.626221 Best loss: 1.509603 Accuracy: 19.27% 7 Validation loss: 1907.869629 Best loss: 1.509603 Accuracy: 22.01% 8 Validation loss: 4393.044434 Best loss: 1.509603 Accuracy: 18.73% 9 Validation loss: 1111.406006 Best loss: 1.509603 Accuracy: 22.79% 10 Validation loss: 1104.229980 Best loss: 1.509603 Accuracy: 19.27% 11 Validation loss: 1206.439331 Best loss: 1.509603 Accuracy: 18.73% 12 Validation loss: 838.631226 Best loss: 1.509603 Accuracy: 20.91% 13 Validation loss: 1299.682739 Best loss: 1.509603 Accuracy: 22.01% 14 Validation loss: 1973.122192 Best loss: 1.509603 Accuracy: 18.73% 15 Validation loss: 2630.484375 Best loss: 1.509603 Accuracy: 18.73% 16 Validation loss: 3163.731934 Best loss: 1.509603 Accuracy: 20.91% 17 Validation loss: 3532.997559 Best loss: 1.509603 Accuracy: 19.27% 18 Validation loss: 4044.567139 Best loss: 1.509603 Accuracy: 22.01% 19 Validation loss: 3543.684814 Best loss: 1.509603 Accuracy: 20.91% 20 Validation loss: 5389.517578 Best loss: 1.509603 Accuracy: 18.73% 21 Validation loss: 674.147095 Best loss: 1.509603 Accuracy: 22.01% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 3.020770 Best loss: 3.020770 Accuracy: 24.16% 1 Validation loss: 18.338531 Best loss: 3.020770 Accuracy: 19.08% 2 Validation loss: 29.121038 Best loss: 3.020770 Accuracy: 18.73% 3 Validation loss: 55.388584 Best loss: 3.020770 Accuracy: 19.19% 4 Validation loss: 52.922451 Best loss: 3.020770 Accuracy: 22.75% 5 Validation loss: 158.213989 Best loss: 3.020770 Accuracy: 22.01% 6 Validation loss: 157.805115 Best loss: 3.020770 Accuracy: 19.27% 7 Validation loss: 201.607361 Best loss: 3.020770 Accuracy: 18.73% 8 Validation loss: 9881.346680 Best loss: 3.020770 Accuracy: 19.08% 9 Validation loss: 1361.788452 Best loss: 3.020770 Accuracy: 19.27% 10 Validation loss: 1854.031128 Best loss: 3.020770 Accuracy: 20.91% 11 Validation loss: 1397.591431 Best loss: 3.020770 Accuracy: 19.27% 12 Validation loss: 1408.930786 Best loss: 3.020770 Accuracy: 18.73% 13 Validation loss: 2691.429443 Best loss: 3.020770 Accuracy: 22.01% 14 Validation loss: 1184.405640 Best loss: 3.020770 Accuracy: 23.06% 15 Validation loss: 1047.187012 Best loss: 3.020770 Accuracy: 22.01% 16 Validation loss: 4306.635742 Best loss: 3.020770 Accuracy: 19.08% 17 Validation loss: 13740.384766 Best loss: 3.020770 Accuracy: 18.73% 18 Validation loss: 2529.918213 Best loss: 3.020770 Accuracy: 19.08% 19 Validation loss: 2409.986816 Best loss: 3.020770 Accuracy: 15.21% 20 Validation loss: 1640.008789 Best loss: 3.020770 Accuracy: 21.23% 21 Validation loss: 13687.430664 Best loss: 3.020770 Accuracy: 18.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=10, learning_rate=0.02, dropout_rate=0.6, total= 2.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.103627 Best loss: 0.103627 Accuracy: 97.15% 1 Validation loss: 0.103066 Best loss: 0.103066 Accuracy: 97.26% 2 Validation loss: 0.127216 Best loss: 0.103066 Accuracy: 97.30% 3 Validation loss: 0.107253 Best loss: 0.103066 Accuracy: 97.54% 4 Validation loss: 1.695790 Best loss: 0.103066 Accuracy: 19.27% 5 Validation loss: 1.615372 Best loss: 0.103066 Accuracy: 20.91% 6 Validation loss: 1.672457 Best loss: 0.103066 Accuracy: 19.27% 7 Validation loss: 1.649497 Best loss: 0.103066 Accuracy: 19.27% 8 Validation loss: 1.677869 Best loss: 0.103066 Accuracy: 22.01% 9 Validation loss: 1.831496 Best loss: 0.103066 Accuracy: 19.08% 10 Validation loss: 1.663474 Best loss: 0.103066 Accuracy: 19.08% 11 Validation loss: 1.663909 Best loss: 0.103066 Accuracy: 18.73% 12 Validation loss: 1.624144 Best loss: 0.103066 Accuracy: 19.08% 13 Validation loss: 1.623486 Best loss: 0.103066 Accuracy: 22.01% 14 Validation loss: 1.653587 Best loss: 0.103066 Accuracy: 19.08% 15 Validation loss: 1.677565 Best loss: 0.103066 Accuracy: 22.01% 16 Validation loss: 1.659621 Best loss: 0.103066 Accuracy: 18.73% 17 Validation loss: 1.681787 Best loss: 0.103066 Accuracy: 20.91% 18 Validation loss: 1.660203 Best loss: 0.103066 Accuracy: 22.01% 19 Validation loss: 1.643599 Best loss: 0.103066 Accuracy: 18.73% 20 Validation loss: 1.681041 Best loss: 0.103066 Accuracy: 19.27% 21 Validation loss: 1.635469 Best loss: 0.103066 Accuracy: 19.08% 22 Validation loss: 1.631423 Best loss: 0.103066 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2, total= 14.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.103414 Best loss: 0.103414 Accuracy: 96.87% 1 Validation loss: 0.104384 Best loss: 0.103414 Accuracy: 97.34% 2 Validation loss: 0.222844 Best loss: 0.103414 Accuracy: 95.23% 3 Validation loss: 1.558907 Best loss: 0.103414 Accuracy: 24.71% 4 Validation loss: 1.711619 Best loss: 0.103414 Accuracy: 19.27% 5 Validation loss: 1.619626 Best loss: 0.103414 Accuracy: 22.01% 6 Validation loss: 1.696654 Best loss: 0.103414 Accuracy: 19.08% 7 Validation loss: 1.626351 Best loss: 0.103414 Accuracy: 19.27% 8 Validation loss: 1.647954 Best loss: 0.103414 Accuracy: 19.27% 9 Validation loss: 1.647573 Best loss: 0.103414 Accuracy: 22.01% 10 Validation loss: 1.650356 Best loss: 0.103414 Accuracy: 20.91% 11 Validation loss: 1.644352 Best loss: 0.103414 Accuracy: 22.01% 12 Validation loss: 1.649301 Best loss: 0.103414 Accuracy: 22.01% 13 Validation loss: 1.645392 Best loss: 0.103414 Accuracy: 18.73% 14 Validation loss: 1.688263 Best loss: 0.103414 Accuracy: 19.08% 15 Validation loss: 1.624780 Best loss: 0.103414 Accuracy: 20.91% 16 Validation loss: 1.653948 Best loss: 0.103414 Accuracy: 19.27% 17 Validation loss: 1.708024 Best loss: 0.103414 Accuracy: 20.91% 18 Validation loss: 1.689577 Best loss: 0.103414 Accuracy: 22.01% 19 Validation loss: 1.627441 Best loss: 0.103414 Accuracy: 18.73% 20 Validation loss: 1.677410 Best loss: 0.103414 Accuracy: 20.91% 21 Validation loss: 1.635521 Best loss: 0.103414 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2, total= 13.4s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.135462 Best loss: 0.135462 Accuracy: 96.72% 1 Validation loss: 0.164250 Best loss: 0.135462 Accuracy: 96.33% 2 Validation loss: 0.108366 Best loss: 0.108366 Accuracy: 97.50% 3 Validation loss: 0.207984 Best loss: 0.108366 Accuracy: 97.97% 4 Validation loss: 1.357586 Best loss: 0.108366 Accuracy: 38.35% 5 Validation loss: 1.696818 Best loss: 0.108366 Accuracy: 22.01% 6 Validation loss: 1.750018 Best loss: 0.108366 Accuracy: 19.27% 7 Validation loss: 1.622157 Best loss: 0.108366 Accuracy: 22.01% 8 Validation loss: 1.649781 Best loss: 0.108366 Accuracy: 18.73% 9 Validation loss: 1.667065 Best loss: 0.108366 Accuracy: 19.27% 10 Validation loss: 1.634763 Best loss: 0.108366 Accuracy: 22.01% 11 Validation loss: 1.658202 Best loss: 0.108366 Accuracy: 19.27% 12 Validation loss: 1.624916 Best loss: 0.108366 Accuracy: 20.91% 13 Validation loss: 1.645393 Best loss: 0.108366 Accuracy: 19.08% 14 Validation loss: 1.667968 Best loss: 0.108366 Accuracy: 18.73% 15 Validation loss: 1.630408 Best loss: 0.108366 Accuracy: 20.91% 16 Validation loss: 1.687083 Best loss: 0.108366 Accuracy: 19.27% 17 Validation loss: 1.657803 Best loss: 0.108366 Accuracy: 19.08% 18 Validation loss: 1.676464 Best loss: 0.108366 Accuracy: 18.73% 19 Validation loss: 1.672037 Best loss: 0.108366 Accuracy: 18.73% 20 Validation loss: 1.648855 Best loss: 0.108366 Accuracy: 19.08% 21 Validation loss: 1.643813 Best loss: 0.108366 Accuracy: 18.73% 22 Validation loss: 1.646201 Best loss: 0.108366 Accuracy: 19.08% 23 Validation loss: 1.659837 Best loss: 0.108366 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=100, batch_size=100, learning_rate=0.02, dropout_rate=0.2, total= 14.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 1.795792 Best loss: 1.795792 Accuracy: 19.27% 1 Validation loss: 1.685615 Best loss: 1.685615 Accuracy: 19.08% 2 Validation loss: 1.891959 Best loss: 1.685615 Accuracy: 22.01% 3 Validation loss: 1.641420 Best loss: 1.641420 Accuracy: 22.01% 4 Validation loss: 1.679689 Best loss: 1.641420 Accuracy: 22.01% 5 Validation loss: 1.762609 Best loss: 1.641420 Accuracy: 19.08% 6 Validation loss: 1.859939 Best loss: 1.641420 Accuracy: 22.01% 7 Validation loss: 1.722116 Best loss: 1.641420 Accuracy: 19.27% 8 Validation loss: 1.796097 Best loss: 1.641420 Accuracy: 19.27% 9 Validation loss: 1.775481 Best loss: 1.641420 Accuracy: 19.08% 10 Validation loss: 1.683851 Best loss: 1.641420 Accuracy: 20.91% 11 Validation loss: 1.747333 Best loss: 1.641420 Accuracy: 19.08% 12 Validation loss: 1.692366 Best loss: 1.641420 Accuracy: 19.08% 13 Validation loss: 1.650157 Best loss: 1.641420 Accuracy: 19.27% 14 Validation loss: 1.938031 Best loss: 1.641420 Accuracy: 18.73% 15 Validation loss: 1.890101 Best loss: 1.641420 Accuracy: 22.01% 16 Validation loss: 1.654253 Best loss: 1.641420 Accuracy: 22.01% 17 Validation loss: 1.671696 Best loss: 1.641420 Accuracy: 19.27% 18 Validation loss: 2.251617 Best loss: 1.641420 Accuracy: 20.91% 19 Validation loss: 1.899849 Best loss: 1.641420 Accuracy: 20.91% 20 Validation loss: 1.640932 Best loss: 1.640932 Accuracy: 22.01% 21 Validation loss: 1.734331 Best loss: 1.640932 Accuracy: 19.27% 22 Validation loss: 1.634685 Best loss: 1.634685 Accuracy: 19.08% 23 Validation loss: 2.018623 Best loss: 1.634685 Accuracy: 22.01% 24 Validation loss: 1.678287 Best loss: 1.634685 Accuracy: 20.91% 25 Validation loss: 1.969434 Best loss: 1.634685 Accuracy: 19.27% 26 Validation loss: 1.725711 Best loss: 1.634685 Accuracy: 19.08% 27 Validation loss: 1.758585 Best loss: 1.634685 Accuracy: 22.01% 28 Validation loss: 1.706162 Best loss: 1.634685 Accuracy: 22.01% 29 Validation loss: 1.817581 Best loss: 1.634685 Accuracy: 18.73% 30 Validation loss: 1.750705 Best loss: 1.634685 Accuracy: 18.73% 31 Validation loss: 1.768409 Best loss: 1.634685 Accuracy: 22.01% 32 Validation loss: 1.941819 Best loss: 1.634685 Accuracy: 18.73% 33 Validation loss: 1.987176 Best loss: 1.634685 Accuracy: 22.01% 34 Validation loss: 1.714575 Best loss: 1.634685 Accuracy: 22.01% 35 Validation loss: 1.903635 Best loss: 1.634685 Accuracy: 19.08% 36 Validation loss: 1.851149 Best loss: 1.634685 Accuracy: 22.01% 37 Validation loss: 1.933181 Best loss: 1.634685 Accuracy: 19.27% 38 Validation loss: 1.927564 Best loss: 1.634685 Accuracy: 18.73% 39 Validation loss: 1.640298 Best loss: 1.634685 Accuracy: 19.08% 40 Validation loss: 1.713870 Best loss: 1.634685 Accuracy: 19.27% 41 Validation loss: 1.823286 Best loss: 1.634685 Accuracy: 20.91% 42 Validation loss: 1.841897 Best loss: 1.634685 Accuracy: 18.73% 43 Validation loss: 1.722634 Best loss: 1.634685 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 25.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 1.863176 Best loss: 1.863176 Accuracy: 20.91% 1 Validation loss: 1.843982 Best loss: 1.843982 Accuracy: 19.08% 2 Validation loss: 1.706580 Best loss: 1.706580 Accuracy: 19.08% 3 Validation loss: 1.690301 Best loss: 1.690301 Accuracy: 22.01% 4 Validation loss: 1.652281 Best loss: 1.652281 Accuracy: 20.91% 5 Validation loss: 1.641099 Best loss: 1.641099 Accuracy: 19.08% 6 Validation loss: 1.758411 Best loss: 1.641099 Accuracy: 19.27% 7 Validation loss: 1.859714 Best loss: 1.641099 Accuracy: 18.73% 8 Validation loss: 1.849591 Best loss: 1.641099 Accuracy: 18.73% 9 Validation loss: 2.009557 Best loss: 1.641099 Accuracy: 19.08% 10 Validation loss: 1.719042 Best loss: 1.641099 Accuracy: 20.91% 11 Validation loss: 1.710112 Best loss: 1.641099 Accuracy: 22.01% 12 Validation loss: 1.841489 Best loss: 1.641099 Accuracy: 20.91% 13 Validation loss: 1.654160 Best loss: 1.641099 Accuracy: 19.08% 14 Validation loss: 1.843194 Best loss: 1.641099 Accuracy: 19.27% 15 Validation loss: 1.667386 Best loss: 1.641099 Accuracy: 22.01% 16 Validation loss: 1.721236 Best loss: 1.641099 Accuracy: 19.08% 17 Validation loss: 1.779953 Best loss: 1.641099 Accuracy: 19.08% 18 Validation loss: 1.829657 Best loss: 1.641099 Accuracy: 20.91% 19 Validation loss: 1.749924 Best loss: 1.641099 Accuracy: 18.73% 20 Validation loss: 1.951008 Best loss: 1.641099 Accuracy: 19.08% 21 Validation loss: 1.744860 Best loss: 1.641099 Accuracy: 22.01% 22 Validation loss: 1.857269 Best loss: 1.641099 Accuracy: 19.08% 23 Validation loss: 1.694803 Best loss: 1.641099 Accuracy: 22.01% 24 Validation loss: 2.048266 Best loss: 1.641099 Accuracy: 19.27% 25 Validation loss: 1.851451 Best loss: 1.641099 Accuracy: 22.01% 26 Validation loss: 1.729969 Best loss: 1.641099 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 15.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 1.882610 Best loss: 1.882610 Accuracy: 22.01% 1 Validation loss: 1.662126 Best loss: 1.662126 Accuracy: 22.01% 2 Validation loss: 1.660540 Best loss: 1.660540 Accuracy: 19.27% 3 Validation loss: 2.037754 Best loss: 1.660540 Accuracy: 19.27% 4 Validation loss: 1.806196 Best loss: 1.660540 Accuracy: 22.01% 5 Validation loss: 1.633937 Best loss: 1.633937 Accuracy: 22.01% 6 Validation loss: 1.725480 Best loss: 1.633937 Accuracy: 19.27% 7 Validation loss: 1.613322 Best loss: 1.613322 Accuracy: 22.01% 8 Validation loss: 1.963393 Best loss: 1.613322 Accuracy: 22.01% 9 Validation loss: 33.561790 Best loss: 1.613322 Accuracy: 20.91% 10 Validation loss: 4.502992 Best loss: 1.613322 Accuracy: 19.08% 11 Validation loss: 8.379320 Best loss: 1.613322 Accuracy: 19.08% 12 Validation loss: 6.122594 Best loss: 1.613322 Accuracy: 20.91% 13 Validation loss: 1.810237 Best loss: 1.613322 Accuracy: 19.08% 14 Validation loss: 2.045719 Best loss: 1.613322 Accuracy: 20.91% 15 Validation loss: 24.525656 Best loss: 1.613322 Accuracy: 22.01% 16 Validation loss: 14.264637 Best loss: 1.613322 Accuracy: 19.27% 17 Validation loss: 1.696099 Best loss: 1.613322 Accuracy: 18.73% 18 Validation loss: 1.611674 Best loss: 1.611674 Accuracy: 22.01% 19 Validation loss: 2.506664 Best loss: 1.611674 Accuracy: 19.08% 20 Validation loss: 1.622596 Best loss: 1.611674 Accuracy: 22.01% 21 Validation loss: 3.034039 Best loss: 1.611674 Accuracy: 19.08% 22 Validation loss: 2.684966 Best loss: 1.611674 Accuracy: 19.08% 23 Validation loss: 1.613771 Best loss: 1.611674 Accuracy: 19.27% 24 Validation loss: 3.621312 Best loss: 1.611674 Accuracy: 20.91% 25 Validation loss: 1.665264 Best loss: 1.611674 Accuracy: 19.27% 26 Validation loss: 1.732466 Best loss: 1.611674 Accuracy: 22.01% 27 Validation loss: 1.633545 Best loss: 1.611674 Accuracy: 19.27% 28 Validation loss: 8.288097 Best loss: 1.611674 Accuracy: 20.91% 29 Validation loss: 61.994514 Best loss: 1.611674 Accuracy: 20.91% 30 Validation loss: 16.840422 Best loss: 1.611674 Accuracy: 20.91% 31 Validation loss: 11.750014 Best loss: 1.611674 Accuracy: 22.01% 32 Validation loss: 5.886500 Best loss: 1.611674 Accuracy: 22.01% 33 Validation loss: 1.615923 Best loss: 1.611674 Accuracy: 19.27% 34 Validation loss: 9.503445 Best loss: 1.611674 Accuracy: 19.27% 35 Validation loss: 2.235021 Best loss: 1.611674 Accuracy: 19.08% 36 Validation loss: 1.609788 Best loss: 1.609788 Accuracy: 20.91% 37 Validation loss: 1.607734 Best loss: 1.607734 Accuracy: 22.01% 38 Validation loss: 1.611388 Best loss: 1.607734 Accuracy: 22.01% 39 Validation loss: 1.611816 Best loss: 1.607734 Accuracy: 19.27% 40 Validation loss: 1.611833 Best loss: 1.607734 Accuracy: 19.08% 41 Validation loss: 1.609873 Best loss: 1.607734 Accuracy: 19.08% 42 Validation loss: 1.609300 Best loss: 1.607734 Accuracy: 22.01% 43 Validation loss: 2.073288 Best loss: 1.607734 Accuracy: 18.73% 44 Validation loss: 1.686911 Best loss: 1.607734 Accuracy: 19.08% 45 Validation loss: 4.860288 Best loss: 1.607734 Accuracy: 19.08% 46 Validation loss: 1.638264 Best loss: 1.607734 Accuracy: 19.27% 47 Validation loss: 1.619758 Best loss: 1.607734 Accuracy: 22.01% 48 Validation loss: 1.611984 Best loss: 1.607734 Accuracy: 18.73% 49 Validation loss: 2.876240 Best loss: 1.607734 Accuracy: 19.27% 50 Validation loss: 1.612755 Best loss: 1.607734 Accuracy: 19.27% 51 Validation loss: 1.616666 Best loss: 1.607734 Accuracy: 22.01% 52 Validation loss: 1.617192 Best loss: 1.607734 Accuracy: 20.91% 53 Validation loss: 1.614530 Best loss: 1.607734 Accuracy: 22.01% 54 Validation loss: 1.612345 Best loss: 1.607734 Accuracy: 22.01% 55 Validation loss: 1.615740 Best loss: 1.607734 Accuracy: 22.01% 56 Validation loss: 1.612412 Best loss: 1.607734 Accuracy: 20.91% 57 Validation loss: 1.635362 Best loss: 1.607734 Accuracy: 18.73% 58 Validation loss: 1.616300 Best loss: 1.607734 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=90, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 34.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.146392 Best loss: 0.146392 Accuracy: 96.56% 1 Validation loss: 0.120429 Best loss: 0.120429 Accuracy: 96.91% 2 Validation loss: 1.489969 Best loss: 0.120429 Accuracy: 33.54% 3 Validation loss: 1.671705 Best loss: 0.120429 Accuracy: 19.08% 4 Validation loss: 1.699266 Best loss: 0.120429 Accuracy: 22.01% 5 Validation loss: 1.624298 Best loss: 0.120429 Accuracy: 18.73% 6 Validation loss: 1.665666 Best loss: 0.120429 Accuracy: 19.27% 7 Validation loss: 1.642542 Best loss: 0.120429 Accuracy: 22.01% 8 Validation loss: 1.710791 Best loss: 0.120429 Accuracy: 19.27% 9 Validation loss: 1.676151 Best loss: 0.120429 Accuracy: 22.01% 10 Validation loss: 1.630410 Best loss: 0.120429 Accuracy: 20.91% 11 Validation loss: 1.628320 Best loss: 0.120429 Accuracy: 22.01% 12 Validation loss: 1.618255 Best loss: 0.120429 Accuracy: 22.01% 13 Validation loss: 1.608927 Best loss: 0.120429 Accuracy: 20.91% 14 Validation loss: 1.647018 Best loss: 0.120429 Accuracy: 19.08% 15 Validation loss: 1.625227 Best loss: 0.120429 Accuracy: 22.01% 16 Validation loss: 1.635628 Best loss: 0.120429 Accuracy: 22.01% 17 Validation loss: 1.628857 Best loss: 0.120429 Accuracy: 19.27% 18 Validation loss: 1.687967 Best loss: 0.120429 Accuracy: 22.01% 19 Validation loss: 1.624529 Best loss: 0.120429 Accuracy: 22.01% 20 Validation loss: 1.651006 Best loss: 0.120429 Accuracy: 19.27% 21 Validation loss: 1.650430 Best loss: 0.120429 Accuracy: 19.08% 22 Validation loss: 1.636513 Best loss: 0.120429 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 13.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.141942 Best loss: 0.141942 Accuracy: 96.05% 1 Validation loss: 0.195724 Best loss: 0.141942 Accuracy: 96.09% 2 Validation loss: 0.119987 Best loss: 0.119987 Accuracy: 96.52% 3 Validation loss: 0.121255 Best loss: 0.119987 Accuracy: 96.52% 4 Validation loss: 1.482739 Best loss: 0.119987 Accuracy: 30.30% 5 Validation loss: 1.649375 Best loss: 0.119987 Accuracy: 22.01% 6 Validation loss: 1.654836 Best loss: 0.119987 Accuracy: 19.08% 7 Validation loss: 1.623386 Best loss: 0.119987 Accuracy: 19.27% 8 Validation loss: 1.637855 Best loss: 0.119987 Accuracy: 19.27% 9 Validation loss: 1.690962 Best loss: 0.119987 Accuracy: 19.27% 10 Validation loss: 1.614113 Best loss: 0.119987 Accuracy: 22.01% 11 Validation loss: 1.638841 Best loss: 0.119987 Accuracy: 22.01% 12 Validation loss: 1.631792 Best loss: 0.119987 Accuracy: 22.01% 13 Validation loss: 1.658727 Best loss: 0.119987 Accuracy: 18.73% 14 Validation loss: 1.633473 Best loss: 0.119987 Accuracy: 22.01% 15 Validation loss: 1.630862 Best loss: 0.119987 Accuracy: 19.08% 16 Validation loss: 1.622334 Best loss: 0.119987 Accuracy: 19.27% 17 Validation loss: 1.642858 Best loss: 0.119987 Accuracy: 20.91% 18 Validation loss: 1.681294 Best loss: 0.119987 Accuracy: 22.01% 19 Validation loss: 1.632692 Best loss: 0.119987 Accuracy: 19.27% 20 Validation loss: 1.629721 Best loss: 0.119987 Accuracy: 18.73% 21 Validation loss: 1.698278 Best loss: 0.119987 Accuracy: 22.01% 22 Validation loss: 1.624502 Best loss: 0.119987 Accuracy: 19.27% 23 Validation loss: 1.695911 Best loss: 0.119987 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 14.4s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.130971 Best loss: 0.130971 Accuracy: 97.15% 1 Validation loss: 0.168928 Best loss: 0.130971 Accuracy: 96.13% 2 Validation loss: 1.492849 Best loss: 0.130971 Accuracy: 39.37% 3 Validation loss: 1.089182 Best loss: 0.130971 Accuracy: 46.76% 4 Validation loss: 1.200720 Best loss: 0.130971 Accuracy: 39.56% 5 Validation loss: 1.419316 Best loss: 0.130971 Accuracy: 35.18% 6 Validation loss: 1.160291 Best loss: 0.130971 Accuracy: 40.34% 7 Validation loss: 1.206214 Best loss: 0.130971 Accuracy: 40.97% 8 Validation loss: 1.258763 Best loss: 0.130971 Accuracy: 39.37% 9 Validation loss: 1.219023 Best loss: 0.130971 Accuracy: 39.33% 10 Validation loss: 1.315330 Best loss: 0.130971 Accuracy: 36.24% 11 Validation loss: 1.493296 Best loss: 0.130971 Accuracy: 30.84% 12 Validation loss: 1.657206 Best loss: 0.130971 Accuracy: 20.60% 13 Validation loss: 1.571080 Best loss: 0.130971 Accuracy: 22.01% 14 Validation loss: 1.587311 Best loss: 0.130971 Accuracy: 23.73% 15 Validation loss: 1.642282 Best loss: 0.130971 Accuracy: 19.08% 16 Validation loss: 1.674813 Best loss: 0.130971 Accuracy: 18.73% 17 Validation loss: 1.655065 Best loss: 0.130971 Accuracy: 20.91% 18 Validation loss: 1.649540 Best loss: 0.130971 Accuracy: 18.73% 19 Validation loss: 1.640884 Best loss: 0.130971 Accuracy: 19.08% 20 Validation loss: 1.640028 Best loss: 0.130971 Accuracy: 19.27% 21 Validation loss: 1.686190 Best loss: 0.130971 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=70, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 13.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 263.247833 Best loss: 263.247833 Accuracy: 18.73% 1 Validation loss: 1098.371826 Best loss: 263.247833 Accuracy: 19.08% 2 Validation loss: 145.998566 Best loss: 145.998566 Accuracy: 19.27% 3 Validation loss: 3912.681396 Best loss: 145.998566 Accuracy: 18.73% 4 Validation loss: 804.942505 Best loss: 145.998566 Accuracy: 18.73% 5 Validation loss: 1844.973999 Best loss: 145.998566 Accuracy: 19.27% 6 Validation loss: 1559.880859 Best loss: 145.998566 Accuracy: 18.73% 7 Validation loss: 5322.495605 Best loss: 145.998566 Accuracy: 22.01% 8 Validation loss: 593.091614 Best loss: 145.998566 Accuracy: 20.99% 9 Validation loss: 325.318451 Best loss: 145.998566 Accuracy: 23.26% 10 Validation loss: 1799.703735 Best loss: 145.998566 Accuracy: 18.73% 11 Validation loss: 1827.461914 Best loss: 145.998566 Accuracy: 18.73% 12 Validation loss: 577.281311 Best loss: 145.998566 Accuracy: 22.01% 13 Validation loss: 2834.160400 Best loss: 145.998566 Accuracy: 19.23% 14 Validation loss: 3264.740967 Best loss: 145.998566 Accuracy: 18.73% 15 Validation loss: 17944.894531 Best loss: 145.998566 Accuracy: 18.73% 16 Validation loss: 14338.915039 Best loss: 145.998566 Accuracy: 19.08% 17 Validation loss: 1052.467773 Best loss: 145.998566 Accuracy: 19.31% 18 Validation loss: 1705.473999 Best loss: 145.998566 Accuracy: 34.79% 19 Validation loss: 1928.342896 Best loss: 145.998566 Accuracy: 22.01% 20 Validation loss: 463.491882 Best loss: 145.998566 Accuracy: 21.58% 21 Validation loss: 281923.562500 Best loss: 145.998566 Accuracy: 20.91% 22 Validation loss: 26149.847656 Best loss: 145.998566 Accuracy: 22.01% 23 Validation loss: 4904.671387 Best loss: 145.998566 Accuracy: 19.08% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 2.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 1409.720215 Best loss: 1409.720215 Accuracy: 22.01% 1 Validation loss: 1852.271118 Best loss: 1409.720215 Accuracy: 32.10% 2 Validation loss: 521.093201 Best loss: 521.093201 Accuracy: 19.27% 3 Validation loss: 481.446503 Best loss: 481.446503 Accuracy: 22.17% 4 Validation loss: 78105.101562 Best loss: 481.446503 Accuracy: 19.08% 5 Validation loss: 1535.206421 Best loss: 481.446503 Accuracy: 20.91% 6 Validation loss: 856.358887 Best loss: 481.446503 Accuracy: 20.91% 7 Validation loss: 3541.211182 Best loss: 481.446503 Accuracy: 19.27% 8 Validation loss: 15650.010742 Best loss: 481.446503 Accuracy: 18.73% 9 Validation loss: 4457.892578 Best loss: 481.446503 Accuracy: 19.27% 10 Validation loss: 17652.625000 Best loss: 481.446503 Accuracy: 20.91% 11 Validation loss: 1645.467529 Best loss: 481.446503 Accuracy: 19.08% 12 Validation loss: 6591.615234 Best loss: 481.446503 Accuracy: 19.27% 13 Validation loss: 6902.818848 Best loss: 481.446503 Accuracy: 20.91% 14 Validation loss: 1514.362061 Best loss: 481.446503 Accuracy: 33.62% 15 Validation loss: 1746.619019 Best loss: 481.446503 Accuracy: 35.50% 16 Validation loss: 939.608704 Best loss: 481.446503 Accuracy: 22.01% 17 Validation loss: 31460.208984 Best loss: 481.446503 Accuracy: 19.08% 18 Validation loss: 3112.698242 Best loss: 481.446503 Accuracy: 19.08% 19 Validation loss: 463.286255 Best loss: 463.286255 Accuracy: 19.27% 20 Validation loss: 18121.964844 Best loss: 463.286255 Accuracy: 19.27% 21 Validation loss: 14702.472656 Best loss: 463.286255 Accuracy: 20.91% 22 Validation loss: 2921.263184 Best loss: 463.286255 Accuracy: 20.91% 23 Validation loss: 10961.828125 Best loss: 463.286255 Accuracy: 19.08% 24 Validation loss: 582.089233 Best loss: 463.286255 Accuracy: 22.01% 25 Validation loss: 13028.379883 Best loss: 463.286255 Accuracy: 19.27% 26 Validation loss: 1175.671631 Best loss: 463.286255 Accuracy: 31.67% 27 Validation loss: 600.702637 Best loss: 463.286255 Accuracy: 22.01% 28 Validation loss: 25608.691406 Best loss: 463.286255 Accuracy: 19.27% 29 Validation loss: 1831.686035 Best loss: 463.286255 Accuracy: 19.08% 30 Validation loss: 6323.765625 Best loss: 463.286255 Accuracy: 19.27% 31 Validation loss: 1419.892456 Best loss: 463.286255 Accuracy: 22.01% 32 Validation loss: 1381.490845 Best loss: 463.286255 Accuracy: 19.12% 33 Validation loss: 22519.400391 Best loss: 463.286255 Accuracy: 20.91% 34 Validation loss: 11330.050781 Best loss: 463.286255 Accuracy: 22.05% 35 Validation loss: 6360.473633 Best loss: 463.286255 Accuracy: 18.76% 36 Validation loss: 702.620483 Best loss: 463.286255 Accuracy: 31.98% 37 Validation loss: 3713.406494 Best loss: 463.286255 Accuracy: 20.91% 38 Validation loss: 891.243042 Best loss: 463.286255 Accuracy: 26.00% 39 Validation loss: 4135.151855 Best loss: 463.286255 Accuracy: 18.73% 40 Validation loss: 6453.114746 Best loss: 463.286255 Accuracy: 18.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 4.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 94.464073 Best loss: 94.464073 Accuracy: 27.17% 1 Validation loss: 21628.197266 Best loss: 94.464073 Accuracy: 18.73% 2 Validation loss: 876.793274 Best loss: 94.464073 Accuracy: 18.73% 3 Validation loss: 5602.726074 Best loss: 94.464073 Accuracy: 20.25% 4 Validation loss: 6890.958008 Best loss: 94.464073 Accuracy: 19.08% 5 Validation loss: 14126.788086 Best loss: 94.464073 Accuracy: 22.01% 6 Validation loss: 841.427307 Best loss: 94.464073 Accuracy: 19.23% 7 Validation loss: 7450.791992 Best loss: 94.464073 Accuracy: 22.17% 8 Validation loss: 74567.796875 Best loss: 94.464073 Accuracy: 18.76% 9 Validation loss: 2379.406250 Best loss: 94.464073 Accuracy: 34.56% 10 Validation loss: 6215.751953 Best loss: 94.464073 Accuracy: 18.30% 11 Validation loss: 1682.017822 Best loss: 94.464073 Accuracy: 20.99% 12 Validation loss: 5647.854492 Best loss: 94.464073 Accuracy: 18.73% 13 Validation loss: 1985.113037 Best loss: 94.464073 Accuracy: 27.48% 14 Validation loss: 1964.915894 Best loss: 94.464073 Accuracy: 26.66% 15 Validation loss: 55348.117188 Best loss: 94.464073 Accuracy: 19.08% 16 Validation loss: 22125.298828 Best loss: 94.464073 Accuracy: 18.73% 17 Validation loss: 94917.617188 Best loss: 94.464073 Accuracy: 22.44% 18 Validation loss: 3118.703369 Best loss: 94.464073 Accuracy: 18.73% 19 Validation loss: 1444.546021 Best loss: 94.464073 Accuracy: 20.91% 20 Validation loss: 2276.637939 Best loss: 94.464073 Accuracy: 20.91% 21 Validation loss: 62866.414062 Best loss: 94.464073 Accuracy: 19.31% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 2.2min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.197093 Best loss: 0.197093 Accuracy: 95.43% 1 Validation loss: 0.212128 Best loss: 0.197093 Accuracy: 95.93% 2 Validation loss: 0.156938 Best loss: 0.156938 Accuracy: 96.01% 3 Validation loss: 0.163646 Best loss: 0.156938 Accuracy: 96.09% 4 Validation loss: 0.159328 Best loss: 0.156938 Accuracy: 95.66% 5 Validation loss: 0.166028 Best loss: 0.156938 Accuracy: 95.86% 6 Validation loss: 0.175159 Best loss: 0.156938 Accuracy: 95.93% 7 Validation loss: 0.198571 Best loss: 0.156938 Accuracy: 96.09% 8 Validation loss: 0.157356 Best loss: 0.156938 Accuracy: 96.21% 9 Validation loss: 0.155921 Best loss: 0.155921 Accuracy: 96.09% 10 Validation loss: 0.142432 Best loss: 0.142432 Accuracy: 96.60% 11 Validation loss: 0.162694 Best loss: 0.142432 Accuracy: 96.05% 12 Validation loss: 0.164627 Best loss: 0.142432 Accuracy: 95.58% 13 Validation loss: 0.170551 Best loss: 0.142432 Accuracy: 96.56% 14 Validation loss: 0.145573 Best loss: 0.142432 Accuracy: 96.29% 15 Validation loss: 0.156696 Best loss: 0.142432 Accuracy: 96.40% 16 Validation loss: 0.154216 Best loss: 0.142432 Accuracy: 95.97% 17 Validation loss: 0.178730 Best loss: 0.142432 Accuracy: 96.21% 18 Validation loss: 0.153894 Best loss: 0.142432 Accuracy: 95.97% 19 Validation loss: 0.174721 Best loss: 0.142432 Accuracy: 96.09% 20 Validation loss: 0.153547 Best loss: 0.142432 Accuracy: 96.48% 21 Validation loss: 0.142800 Best loss: 0.142432 Accuracy: 96.17% 22 Validation loss: 0.160606 Best loss: 0.142432 Accuracy: 96.52% 23 Validation loss: 0.187086 Best loss: 0.142432 Accuracy: 95.31% 24 Validation loss: 0.148527 Best loss: 0.142432 Accuracy: 96.56% 25 Validation loss: 0.187356 Best loss: 0.142432 Accuracy: 96.13% 26 Validation loss: 0.146063 Best loss: 0.142432 Accuracy: 96.60% 27 Validation loss: 0.149562 Best loss: 0.142432 Accuracy: 95.90% 28 Validation loss: 0.161389 Best loss: 0.142432 Accuracy: 95.58% 29 Validation loss: 0.155563 Best loss: 0.142432 Accuracy: 95.93% 30 Validation loss: 0.160791 Best loss: 0.142432 Accuracy: 96.01% 31 Validation loss: 0.160633 Best loss: 0.142432 Accuracy: 96.56% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 36.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.226065 Best loss: 0.226065 Accuracy: 95.23% 1 Validation loss: 0.199991 Best loss: 0.199991 Accuracy: 95.23% 2 Validation loss: 0.179017 Best loss: 0.179017 Accuracy: 95.31% 3 Validation loss: 0.173179 Best loss: 0.173179 Accuracy: 95.82% 4 Validation loss: 0.184635 Best loss: 0.173179 Accuracy: 95.58% 5 Validation loss: 0.186025 Best loss: 0.173179 Accuracy: 95.47% 6 Validation loss: 0.158372 Best loss: 0.158372 Accuracy: 95.82% 7 Validation loss: 0.142790 Best loss: 0.142790 Accuracy: 96.21% 8 Validation loss: 0.162356 Best loss: 0.142790 Accuracy: 95.86% 9 Validation loss: 0.153055 Best loss: 0.142790 Accuracy: 95.86% 10 Validation loss: 0.154145 Best loss: 0.142790 Accuracy: 96.05% 11 Validation loss: 0.147152 Best loss: 0.142790 Accuracy: 95.97% 12 Validation loss: 0.141207 Best loss: 0.141207 Accuracy: 96.29% 13 Validation loss: 0.152858 Best loss: 0.141207 Accuracy: 96.17% 14 Validation loss: 0.171814 Best loss: 0.141207 Accuracy: 96.01% 15 Validation loss: 0.156381 Best loss: 0.141207 Accuracy: 95.97% 16 Validation loss: 0.166175 Best loss: 0.141207 Accuracy: 96.44% 17 Validation loss: 0.186785 Best loss: 0.141207 Accuracy: 95.74% 18 Validation loss: 0.148992 Best loss: 0.141207 Accuracy: 96.33% 19 Validation loss: 0.168807 Best loss: 0.141207 Accuracy: 95.93% 20 Validation loss: 0.179733 Best loss: 0.141207 Accuracy: 95.78% 21 Validation loss: 0.175318 Best loss: 0.141207 Accuracy: 95.23% 22 Validation loss: 0.228519 Best loss: 0.141207 Accuracy: 95.00% 23 Validation loss: 0.164182 Best loss: 0.141207 Accuracy: 96.09% 24 Validation loss: 0.218587 Best loss: 0.141207 Accuracy: 95.70% 25 Validation loss: 0.164407 Best loss: 0.141207 Accuracy: 96.21% 26 Validation loss: 0.161365 Best loss: 0.141207 Accuracy: 95.86% 27 Validation loss: 0.159358 Best loss: 0.141207 Accuracy: 96.21% 28 Validation loss: 0.179121 Best loss: 0.141207 Accuracy: 95.78% 29 Validation loss: 0.175067 Best loss: 0.141207 Accuracy: 96.05% 30 Validation loss: 0.157268 Best loss: 0.141207 Accuracy: 96.13% 31 Validation loss: 0.194328 Best loss: 0.141207 Accuracy: 96.25% 32 Validation loss: 0.156200 Best loss: 0.141207 Accuracy: 95.35% 33 Validation loss: 0.162132 Best loss: 0.141207 Accuracy: 95.90% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 38.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.174860 Best loss: 0.174860 Accuracy: 95.47% 1 Validation loss: 0.168022 Best loss: 0.168022 Accuracy: 95.93% 2 Validation loss: 0.137462 Best loss: 0.137462 Accuracy: 96.68% 3 Validation loss: 0.158907 Best loss: 0.137462 Accuracy: 96.29% 4 Validation loss: 0.150209 Best loss: 0.137462 Accuracy: 96.09% 5 Validation loss: 0.157066 Best loss: 0.137462 Accuracy: 95.70% 6 Validation loss: 0.143018 Best loss: 0.137462 Accuracy: 96.40% 7 Validation loss: 0.137549 Best loss: 0.137462 Accuracy: 96.40% 8 Validation loss: 0.137856 Best loss: 0.137462 Accuracy: 96.56% 9 Validation loss: 0.145162 Best loss: 0.137462 Accuracy: 96.13% 10 Validation loss: 0.157063 Best loss: 0.137462 Accuracy: 96.33% 11 Validation loss: 0.144234 Best loss: 0.137462 Accuracy: 96.56% 12 Validation loss: 0.151235 Best loss: 0.137462 Accuracy: 96.25% 13 Validation loss: 0.140185 Best loss: 0.137462 Accuracy: 96.72% 14 Validation loss: 0.150951 Best loss: 0.137462 Accuracy: 96.13% 15 Validation loss: 0.150693 Best loss: 0.137462 Accuracy: 96.72% 16 Validation loss: 0.159469 Best loss: 0.137462 Accuracy: 95.62% 17 Validation loss: 0.167522 Best loss: 0.137462 Accuracy: 96.21% 18 Validation loss: 0.157048 Best loss: 0.137462 Accuracy: 95.97% 19 Validation loss: 0.155610 Best loss: 0.137462 Accuracy: 96.29% 20 Validation loss: 0.142110 Best loss: 0.137462 Accuracy: 96.25% 21 Validation loss: 0.174993 Best loss: 0.137462 Accuracy: 96.33% 22 Validation loss: 0.164534 Best loss: 0.137462 Accuracy: 96.25% 23 Validation loss: 0.144287 Best loss: 0.137462 Accuracy: 96.48% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 26.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.473047 Best loss: 0.473047 Accuracy: 74.51% 1 Validation loss: 0.289441 Best loss: 0.289441 Accuracy: 92.22% 2 Validation loss: 0.209517 Best loss: 0.209517 Accuracy: 94.80% 3 Validation loss: 0.240829 Best loss: 0.209517 Accuracy: 93.75% 4 Validation loss: 0.168524 Best loss: 0.168524 Accuracy: 95.15% 5 Validation loss: 0.153497 Best loss: 0.153497 Accuracy: 95.47% 6 Validation loss: 0.168429 Best loss: 0.153497 Accuracy: 94.72% 7 Validation loss: 0.157845 Best loss: 0.153497 Accuracy: 95.35% 8 Validation loss: 0.150680 Best loss: 0.150680 Accuracy: 95.74% 9 Validation loss: 0.141331 Best loss: 0.141331 Accuracy: 95.78% 10 Validation loss: 0.144186 Best loss: 0.141331 Accuracy: 96.25% 11 Validation loss: 0.142132 Best loss: 0.141331 Accuracy: 95.93% 12 Validation loss: 0.148011 Best loss: 0.141331 Accuracy: 95.93% 13 Validation loss: 0.142805 Best loss: 0.141331 Accuracy: 95.82% 14 Validation loss: 0.125881 Best loss: 0.125881 Accuracy: 96.29% 15 Validation loss: 0.139359 Best loss: 0.125881 Accuracy: 95.74% 16 Validation loss: 0.136829 Best loss: 0.125881 Accuracy: 95.93% 17 Validation loss: 0.135349 Best loss: 0.125881 Accuracy: 96.44% 18 Validation loss: 0.116592 Best loss: 0.116592 Accuracy: 96.60% 19 Validation loss: 0.122592 Best loss: 0.116592 Accuracy: 96.64% 20 Validation loss: 0.118756 Best loss: 0.116592 Accuracy: 96.72% 21 Validation loss: 0.126447 Best loss: 0.116592 Accuracy: 96.48% 22 Validation loss: 0.131449 Best loss: 0.116592 Accuracy: 96.40% 23 Validation loss: 0.126625 Best loss: 0.116592 Accuracy: 96.76% 24 Validation loss: 0.124501 Best loss: 0.116592 Accuracy: 96.56% 25 Validation loss: 0.141677 Best loss: 0.116592 Accuracy: 95.93% 26 Validation loss: 0.121354 Best loss: 0.116592 Accuracy: 96.44% 27 Validation loss: 0.128495 Best loss: 0.116592 Accuracy: 96.29% 28 Validation loss: 0.135320 Best loss: 0.116592 Accuracy: 96.36% 29 Validation loss: 0.118717 Best loss: 0.116592 Accuracy: 96.52% 30 Validation loss: 0.124034 Best loss: 0.116592 Accuracy: 96.33% 31 Validation loss: 0.124398 Best loss: 0.116592 Accuracy: 96.64% 32 Validation loss: 0.122924 Best loss: 0.116592 Accuracy: 96.68% 33 Validation loss: 0.123838 Best loss: 0.116592 Accuracy: 96.17% 34 Validation loss: 0.135996 Best loss: 0.116592 Accuracy: 96.33% 35 Validation loss: 0.130254 Best loss: 0.116592 Accuracy: 96.60% 36 Validation loss: 0.117164 Best loss: 0.116592 Accuracy: 96.56% 37 Validation loss: 0.116561 Best loss: 0.116561 Accuracy: 96.83% 38 Validation loss: 0.112411 Best loss: 0.112411 Accuracy: 96.99% 39 Validation loss: 0.122853 Best loss: 0.112411 Accuracy: 96.76% 40 Validation loss: 0.126149 Best loss: 0.112411 Accuracy: 96.48% 41 Validation loss: 0.130254 Best loss: 0.112411 Accuracy: 96.56% 42 Validation loss: 0.129680 Best loss: 0.112411 Accuracy: 96.52% 43 Validation loss: 0.123953 Best loss: 0.112411 Accuracy: 96.40% 44 Validation loss: 0.116477 Best loss: 0.112411 Accuracy: 96.56% 45 Validation loss: 0.123820 Best loss: 0.112411 Accuracy: 96.83% 46 Validation loss: 0.116332 Best loss: 0.112411 Accuracy: 96.60% 47 Validation loss: 0.126929 Best loss: 0.112411 Accuracy: 96.44% 48 Validation loss: 0.127661 Best loss: 0.112411 Accuracy: 96.72% 49 Validation loss: 0.131759 Best loss: 0.112411 Accuracy: 96.83% 50 Validation loss: 0.122109 Best loss: 0.112411 Accuracy: 96.87% 51 Validation loss: 0.121560 Best loss: 0.112411 Accuracy: 96.29% 52 Validation loss: 0.122108 Best loss: 0.112411 Accuracy: 96.21% 53 Validation loss: 0.115756 Best loss: 0.112411 Accuracy: 96.68% 54 Validation loss: 0.133051 Best loss: 0.112411 Accuracy: 96.68% 55 Validation loss: 0.118102 Best loss: 0.112411 Accuracy: 96.87% 56 Validation loss: 0.117886 Best loss: 0.112411 Accuracy: 96.95% 57 Validation loss: 0.123407 Best loss: 0.112411 Accuracy: 96.56% 58 Validation loss: 0.126736 Best loss: 0.112411 Accuracy: 96.79% 59 Validation loss: 0.128210 Best loss: 0.112411 Accuracy: 96.72% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 40.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.299613 Best loss: 0.299613 Accuracy: 93.67% 1 Validation loss: 0.180864 Best loss: 0.180864 Accuracy: 95.74% 2 Validation loss: 0.206209 Best loss: 0.180864 Accuracy: 94.57% 3 Validation loss: 0.156989 Best loss: 0.156989 Accuracy: 96.13% 4 Validation loss: 0.161490 Best loss: 0.156989 Accuracy: 95.47% 5 Validation loss: 0.172551 Best loss: 0.156989 Accuracy: 95.43% 6 Validation loss: 0.144249 Best loss: 0.144249 Accuracy: 96.29% 7 Validation loss: 0.148403 Best loss: 0.144249 Accuracy: 96.52% 8 Validation loss: 0.162775 Best loss: 0.144249 Accuracy: 95.66% 9 Validation loss: 0.142110 Best loss: 0.142110 Accuracy: 96.05% 10 Validation loss: 0.157819 Best loss: 0.142110 Accuracy: 95.74% 11 Validation loss: 0.136062 Best loss: 0.136062 Accuracy: 96.21% 12 Validation loss: 0.145168 Best loss: 0.136062 Accuracy: 95.70% 13 Validation loss: 0.130874 Best loss: 0.130874 Accuracy: 96.48% 14 Validation loss: 0.138004 Best loss: 0.130874 Accuracy: 96.91% 15 Validation loss: 0.128456 Best loss: 0.128456 Accuracy: 97.03% 16 Validation loss: 0.126556 Best loss: 0.126556 Accuracy: 96.64% 17 Validation loss: 0.129317 Best loss: 0.126556 Accuracy: 96.83% 18 Validation loss: 0.129481 Best loss: 0.126556 Accuracy: 96.13% 19 Validation loss: 0.139169 Best loss: 0.126556 Accuracy: 96.60% 20 Validation loss: 0.142886 Best loss: 0.126556 Accuracy: 96.25% 21 Validation loss: 0.131126 Best loss: 0.126556 Accuracy: 96.40% 22 Validation loss: 0.133531 Best loss: 0.126556 Accuracy: 96.48% 23 Validation loss: 0.117047 Best loss: 0.117047 Accuracy: 97.11% 24 Validation loss: 0.116976 Best loss: 0.116976 Accuracy: 96.91% 25 Validation loss: 0.123544 Best loss: 0.116976 Accuracy: 96.76% 26 Validation loss: 0.122160 Best loss: 0.116976 Accuracy: 96.99% 27 Validation loss: 0.122392 Best loss: 0.116976 Accuracy: 96.79% 28 Validation loss: 0.115914 Best loss: 0.115914 Accuracy: 96.87% 29 Validation loss: 0.120437 Best loss: 0.115914 Accuracy: 96.52% 30 Validation loss: 0.113597 Best loss: 0.113597 Accuracy: 97.07% 31 Validation loss: 0.124878 Best loss: 0.113597 Accuracy: 96.79% 32 Validation loss: 0.126071 Best loss: 0.113597 Accuracy: 96.95% 33 Validation loss: 0.119377 Best loss: 0.113597 Accuracy: 96.95% 34 Validation loss: 0.121540 Best loss: 0.113597 Accuracy: 96.83% 35 Validation loss: 0.117338 Best loss: 0.113597 Accuracy: 96.72% 36 Validation loss: 0.122250 Best loss: 0.113597 Accuracy: 96.87% 37 Validation loss: 0.116677 Best loss: 0.113597 Accuracy: 97.30% 38 Validation loss: 0.130091 Best loss: 0.113597 Accuracy: 96.48% 39 Validation loss: 0.109522 Best loss: 0.109522 Accuracy: 96.83% 40 Validation loss: 0.115390 Best loss: 0.109522 Accuracy: 96.99% 41 Validation loss: 0.128358 Best loss: 0.109522 Accuracy: 96.21% 42 Validation loss: 0.117603 Best loss: 0.109522 Accuracy: 96.83% 43 Validation loss: 0.130624 Best loss: 0.109522 Accuracy: 96.76% 44 Validation loss: 0.128196 Best loss: 0.109522 Accuracy: 96.48% 45 Validation loss: 0.126682 Best loss: 0.109522 Accuracy: 96.36% 46 Validation loss: 0.116817 Best loss: 0.109522 Accuracy: 96.83% 47 Validation loss: 0.126421 Best loss: 0.109522 Accuracy: 96.83% 48 Validation loss: 0.114368 Best loss: 0.109522 Accuracy: 97.11% 49 Validation loss: 0.119183 Best loss: 0.109522 Accuracy: 96.79% 50 Validation loss: 0.117472 Best loss: 0.109522 Accuracy: 96.68% 51 Validation loss: 0.127759 Best loss: 0.109522 Accuracy: 96.95% 52 Validation loss: 0.121583 Best loss: 0.109522 Accuracy: 96.76% 53 Validation loss: 0.121574 Best loss: 0.109522 Accuracy: 97.03% 54 Validation loss: 0.138861 Best loss: 0.109522 Accuracy: 96.44% 55 Validation loss: 0.118870 Best loss: 0.109522 Accuracy: 96.76% 56 Validation loss: 0.124138 Best loss: 0.109522 Accuracy: 96.60% 57 Validation loss: 0.112318 Best loss: 0.109522 Accuracy: 96.64% 58 Validation loss: 0.116581 Best loss: 0.109522 Accuracy: 97.07% 59 Validation loss: 0.115262 Best loss: 0.109522 Accuracy: 96.95% 60 Validation loss: 0.117546 Best loss: 0.109522 Accuracy: 96.79% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 41.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.615695 Best loss: 0.615695 Accuracy: 71.03% 1 Validation loss: 0.504203 Best loss: 0.504203 Accuracy: 76.35% 2 Validation loss: 0.387438 Best loss: 0.387438 Accuracy: 89.80% 3 Validation loss: 0.296380 Best loss: 0.296380 Accuracy: 91.75% 4 Validation loss: 0.223281 Best loss: 0.223281 Accuracy: 94.02% 5 Validation loss: 0.182393 Best loss: 0.182393 Accuracy: 95.47% 6 Validation loss: 0.179828 Best loss: 0.179828 Accuracy: 95.66% 7 Validation loss: 0.159608 Best loss: 0.159608 Accuracy: 96.25% 8 Validation loss: 0.181821 Best loss: 0.159608 Accuracy: 96.13% 9 Validation loss: 0.159519 Best loss: 0.159519 Accuracy: 96.09% 10 Validation loss: 0.179796 Best loss: 0.159519 Accuracy: 95.47% 11 Validation loss: 0.182992 Best loss: 0.159519 Accuracy: 95.58% 12 Validation loss: 0.162248 Best loss: 0.159519 Accuracy: 96.25% 13 Validation loss: 0.155570 Best loss: 0.155570 Accuracy: 96.79% 14 Validation loss: 0.166507 Best loss: 0.155570 Accuracy: 96.17% 15 Validation loss: 0.164080 Best loss: 0.155570 Accuracy: 96.52% 16 Validation loss: 0.145148 Best loss: 0.145148 Accuracy: 96.40% 17 Validation loss: 0.152768 Best loss: 0.145148 Accuracy: 96.72% 18 Validation loss: 0.164434 Best loss: 0.145148 Accuracy: 95.82% 19 Validation loss: 0.161579 Best loss: 0.145148 Accuracy: 96.40% 20 Validation loss: 0.163114 Best loss: 0.145148 Accuracy: 96.52% 21 Validation loss: 0.149843 Best loss: 0.145148 Accuracy: 96.56% 22 Validation loss: 0.164340 Best loss: 0.145148 Accuracy: 96.40% 23 Validation loss: 0.159801 Best loss: 0.145148 Accuracy: 96.13% 24 Validation loss: 0.158052 Best loss: 0.145148 Accuracy: 96.21% 25 Validation loss: 0.145139 Best loss: 0.145139 Accuracy: 96.36% 26 Validation loss: 0.143346 Best loss: 0.143346 Accuracy: 96.36% 27 Validation loss: 0.146434 Best loss: 0.143346 Accuracy: 96.13% 28 Validation loss: 0.147449 Best loss: 0.143346 Accuracy: 96.33% 29 Validation loss: 0.162573 Best loss: 0.143346 Accuracy: 95.93% 30 Validation loss: 0.150346 Best loss: 0.143346 Accuracy: 96.33% 31 Validation loss: 0.136949 Best loss: 0.136949 Accuracy: 96.48% 32 Validation loss: 0.140772 Best loss: 0.136949 Accuracy: 96.48% 33 Validation loss: 0.143155 Best loss: 0.136949 Accuracy: 96.76% 34 Validation loss: 0.137904 Best loss: 0.136949 Accuracy: 96.79% 35 Validation loss: 0.130557 Best loss: 0.130557 Accuracy: 96.83% 36 Validation loss: 0.131747 Best loss: 0.130557 Accuracy: 96.52% 37 Validation loss: 0.140358 Best loss: 0.130557 Accuracy: 96.64% 38 Validation loss: 0.133335 Best loss: 0.130557 Accuracy: 96.68% 39 Validation loss: 0.127629 Best loss: 0.127629 Accuracy: 96.83% 40 Validation loss: 0.132761 Best loss: 0.127629 Accuracy: 97.03% 41 Validation loss: 0.139449 Best loss: 0.127629 Accuracy: 96.44% 42 Validation loss: 0.130354 Best loss: 0.127629 Accuracy: 97.03% 43 Validation loss: 0.129758 Best loss: 0.127629 Accuracy: 96.95% 44 Validation loss: 0.131549 Best loss: 0.127629 Accuracy: 97.07% 45 Validation loss: 0.132940 Best loss: 0.127629 Accuracy: 96.60% 46 Validation loss: 0.127382 Best loss: 0.127382 Accuracy: 96.87% 47 Validation loss: 0.134154 Best loss: 0.127382 Accuracy: 96.79% 48 Validation loss: 0.123768 Best loss: 0.123768 Accuracy: 96.87% 49 Validation loss: 0.120958 Best loss: 0.120958 Accuracy: 96.91% 50 Validation loss: 0.130382 Best loss: 0.120958 Accuracy: 96.87% 51 Validation loss: 0.128953 Best loss: 0.120958 Accuracy: 96.95% 52 Validation loss: 0.135356 Best loss: 0.120958 Accuracy: 96.87% 53 Validation loss: 0.135401 Best loss: 0.120958 Accuracy: 96.91% 54 Validation loss: 0.121535 Best loss: 0.120958 Accuracy: 97.07% 55 Validation loss: 0.138077 Best loss: 0.120958 Accuracy: 96.56% 56 Validation loss: 0.122534 Best loss: 0.120958 Accuracy: 96.91% 57 Validation loss: 0.129885 Best loss: 0.120958 Accuracy: 96.44% 58 Validation loss: 0.123740 Best loss: 0.120958 Accuracy: 96.72% 59 Validation loss: 0.143259 Best loss: 0.120958 Accuracy: 96.48% 60 Validation loss: 0.147583 Best loss: 0.120958 Accuracy: 96.79% 61 Validation loss: 0.113539 Best loss: 0.113539 Accuracy: 97.07% 62 Validation loss: 0.118385 Best loss: 0.113539 Accuracy: 97.11% 63 Validation loss: 0.115487 Best loss: 0.113539 Accuracy: 97.30% 64 Validation loss: 0.132042 Best loss: 0.113539 Accuracy: 96.91% 65 Validation loss: 0.132528 Best loss: 0.113539 Accuracy: 97.03% 66 Validation loss: 0.126087 Best loss: 0.113539 Accuracy: 97.11% 67 Validation loss: 0.118013 Best loss: 0.113539 Accuracy: 97.22% 68 Validation loss: 0.121197 Best loss: 0.113539 Accuracy: 96.83% 69 Validation loss: 0.128683 Best loss: 0.113539 Accuracy: 96.72% 70 Validation loss: 0.124496 Best loss: 0.113539 Accuracy: 96.83% 71 Validation loss: 0.129623 Best loss: 0.113539 Accuracy: 96.72% 72 Validation loss: 0.130789 Best loss: 0.113539 Accuracy: 96.52% 73 Validation loss: 0.132629 Best loss: 0.113539 Accuracy: 96.56% 74 Validation loss: 0.129877 Best loss: 0.113539 Accuracy: 96.60% 75 Validation loss: 0.124965 Best loss: 0.113539 Accuracy: 97.03% 76 Validation loss: 0.122313 Best loss: 0.113539 Accuracy: 97.07% 77 Validation loss: 0.128417 Best loss: 0.113539 Accuracy: 97.03% 78 Validation loss: 0.135817 Best loss: 0.113539 Accuracy: 96.52% 79 Validation loss: 0.138021 Best loss: 0.113539 Accuracy: 96.56% 80 Validation loss: 0.128907 Best loss: 0.113539 Accuracy: 96.79% 81 Validation loss: 0.129494 Best loss: 0.113539 Accuracy: 96.64% 82 Validation loss: 0.127025 Best loss: 0.113539 Accuracy: 96.76% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=10, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 55.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.129036 Best loss: 0.129036 Accuracy: 96.17% 1 Validation loss: 0.118280 Best loss: 0.118280 Accuracy: 96.36% 2 Validation loss: 0.102820 Best loss: 0.102820 Accuracy: 97.34% 3 Validation loss: 0.095526 Best loss: 0.095526 Accuracy: 97.42% 4 Validation loss: 0.078055 Best loss: 0.078055 Accuracy: 98.12% 5 Validation loss: 0.082150 Best loss: 0.078055 Accuracy: 97.46% 6 Validation loss: 0.076424 Best loss: 0.076424 Accuracy: 97.85% 7 Validation loss: 0.072279 Best loss: 0.072279 Accuracy: 98.28% 8 Validation loss: 0.076117 Best loss: 0.072279 Accuracy: 98.12% 9 Validation loss: 0.075575 Best loss: 0.072279 Accuracy: 98.12% 10 Validation loss: 0.072635 Best loss: 0.072279 Accuracy: 98.24% 11 Validation loss: 0.072431 Best loss: 0.072279 Accuracy: 98.32% 12 Validation loss: 0.064735 Best loss: 0.064735 Accuracy: 98.36% 13 Validation loss: 0.074800 Best loss: 0.064735 Accuracy: 98.12% 14 Validation loss: 0.069109 Best loss: 0.064735 Accuracy: 98.40% 15 Validation loss: 0.069678 Best loss: 0.064735 Accuracy: 98.20% 16 Validation loss: 0.067513 Best loss: 0.064735 Accuracy: 98.24% 17 Validation loss: 0.075438 Best loss: 0.064735 Accuracy: 98.08% 18 Validation loss: 0.071234 Best loss: 0.064735 Accuracy: 98.24% 19 Validation loss: 0.068914 Best loss: 0.064735 Accuracy: 98.48% 20 Validation loss: 0.065232 Best loss: 0.064735 Accuracy: 98.40% 21 Validation loss: 0.067130 Best loss: 0.064735 Accuracy: 98.36% 22 Validation loss: 0.064534 Best loss: 0.064534 Accuracy: 98.48% 23 Validation loss: 0.066065 Best loss: 0.064534 Accuracy: 98.28% 24 Validation loss: 0.065571 Best loss: 0.064534 Accuracy: 98.67% 25 Validation loss: 0.066537 Best loss: 0.064534 Accuracy: 98.24% 26 Validation loss: 0.061435 Best loss: 0.061435 Accuracy: 98.48% 27 Validation loss: 0.067817 Best loss: 0.061435 Accuracy: 98.32% 28 Validation loss: 0.066818 Best loss: 0.061435 Accuracy: 98.36% 29 Validation loss: 0.066245 Best loss: 0.061435 Accuracy: 98.51% 30 Validation loss: 0.067100 Best loss: 0.061435 Accuracy: 98.28% 31 Validation loss: 0.055458 Best loss: 0.055458 Accuracy: 98.55% 32 Validation loss: 0.057453 Best loss: 0.055458 Accuracy: 98.20% 33 Validation loss: 0.063770 Best loss: 0.055458 Accuracy: 98.55% 34 Validation loss: 0.059738 Best loss: 0.055458 Accuracy: 98.51% 35 Validation loss: 0.073756 Best loss: 0.055458 Accuracy: 98.32% 36 Validation loss: 0.063845 Best loss: 0.055458 Accuracy: 98.48% 37 Validation loss: 0.062173 Best loss: 0.055458 Accuracy: 98.55% 38 Validation loss: 0.066794 Best loss: 0.055458 Accuracy: 98.24% 39 Validation loss: 0.064230 Best loss: 0.055458 Accuracy: 98.36% 40 Validation loss: 0.064585 Best loss: 0.055458 Accuracy: 98.75% 41 Validation loss: 0.060063 Best loss: 0.055458 Accuracy: 98.40% 42 Validation loss: 0.065743 Best loss: 0.055458 Accuracy: 98.59% 43 Validation loss: 0.063479 Best loss: 0.055458 Accuracy: 98.59% 44 Validation loss: 0.065298 Best loss: 0.055458 Accuracy: 98.36% 45 Validation loss: 0.067334 Best loss: 0.055458 Accuracy: 98.48% 46 Validation loss: 0.058662 Best loss: 0.055458 Accuracy: 98.51% 47 Validation loss: 0.058689 Best loss: 0.055458 Accuracy: 98.51% 48 Validation loss: 0.066238 Best loss: 0.055458 Accuracy: 98.48% 49 Validation loss: 0.059088 Best loss: 0.055458 Accuracy: 98.55% 50 Validation loss: 0.067450 Best loss: 0.055458 Accuracy: 98.59% 51 Validation loss: 0.058081 Best loss: 0.055458 Accuracy: 98.44% 52 Validation loss: 0.055482 Best loss: 0.055458 Accuracy: 98.32% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4, total= 10.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.145908 Best loss: 0.145908 Accuracy: 95.62% 1 Validation loss: 0.108691 Best loss: 0.108691 Accuracy: 96.48% 2 Validation loss: 0.090648 Best loss: 0.090648 Accuracy: 97.54% 3 Validation loss: 0.085379 Best loss: 0.085379 Accuracy: 97.46% 4 Validation loss: 0.087020 Best loss: 0.085379 Accuracy: 97.69% 5 Validation loss: 0.076088 Best loss: 0.076088 Accuracy: 97.69% 6 Validation loss: 0.067550 Best loss: 0.067550 Accuracy: 98.08% 7 Validation loss: 0.074082 Best loss: 0.067550 Accuracy: 97.65% 8 Validation loss: 0.070841 Best loss: 0.067550 Accuracy: 98.01% 9 Validation loss: 0.072801 Best loss: 0.067550 Accuracy: 98.08% 10 Validation loss: 0.069346 Best loss: 0.067550 Accuracy: 98.32% 11 Validation loss: 0.065744 Best loss: 0.065744 Accuracy: 98.40% 12 Validation loss: 0.066812 Best loss: 0.065744 Accuracy: 98.20% 13 Validation loss: 0.063086 Best loss: 0.063086 Accuracy: 98.24% 14 Validation loss: 0.062580 Best loss: 0.062580 Accuracy: 98.32% 15 Validation loss: 0.068038 Best loss: 0.062580 Accuracy: 98.40% 16 Validation loss: 0.066402 Best loss: 0.062580 Accuracy: 98.28% 17 Validation loss: 0.058631 Best loss: 0.058631 Accuracy: 98.32% 18 Validation loss: 0.062346 Best loss: 0.058631 Accuracy: 98.24% 19 Validation loss: 0.056410 Best loss: 0.056410 Accuracy: 98.51% 20 Validation loss: 0.068669 Best loss: 0.056410 Accuracy: 98.40% 21 Validation loss: 0.055236 Best loss: 0.055236 Accuracy: 98.44% 22 Validation loss: 0.056572 Best loss: 0.055236 Accuracy: 98.40% 23 Validation loss: 0.064847 Best loss: 0.055236 Accuracy: 98.36% 24 Validation loss: 0.055604 Best loss: 0.055236 Accuracy: 98.63% 25 Validation loss: 0.065691 Best loss: 0.055236 Accuracy: 98.20% 26 Validation loss: 0.058653 Best loss: 0.055236 Accuracy: 98.40% 27 Validation loss: 0.065796 Best loss: 0.055236 Accuracy: 98.32% 28 Validation loss: 0.055645 Best loss: 0.055236 Accuracy: 98.48% 29 Validation loss: 0.058963 Best loss: 0.055236 Accuracy: 98.48% 30 Validation loss: 0.055461 Best loss: 0.055236 Accuracy: 98.67% 31 Validation loss: 0.057487 Best loss: 0.055236 Accuracy: 98.67% 32 Validation loss: 0.058932 Best loss: 0.055236 Accuracy: 98.48% 33 Validation loss: 0.060556 Best loss: 0.055236 Accuracy: 98.51% 34 Validation loss: 0.057588 Best loss: 0.055236 Accuracy: 98.48% 35 Validation loss: 0.066379 Best loss: 0.055236 Accuracy: 98.24% 36 Validation loss: 0.059073 Best loss: 0.055236 Accuracy: 98.51% 37 Validation loss: 0.060273 Best loss: 0.055236 Accuracy: 98.28% 38 Validation loss: 0.058596 Best loss: 0.055236 Accuracy: 98.40% 39 Validation loss: 0.063359 Best loss: 0.055236 Accuracy: 98.40% 40 Validation loss: 0.059058 Best loss: 0.055236 Accuracy: 98.51% 41 Validation loss: 0.056185 Best loss: 0.055236 Accuracy: 98.51% 42 Validation loss: 0.060815 Best loss: 0.055236 Accuracy: 98.55% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4, total= 8.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.133376 Best loss: 0.133376 Accuracy: 96.60% 1 Validation loss: 0.104300 Best loss: 0.104300 Accuracy: 97.30% 2 Validation loss: 0.101006 Best loss: 0.101006 Accuracy: 97.19% 3 Validation loss: 0.081340 Best loss: 0.081340 Accuracy: 97.77% 4 Validation loss: 0.099138 Best loss: 0.081340 Accuracy: 97.54% 5 Validation loss: 0.084920 Best loss: 0.081340 Accuracy: 97.81% 6 Validation loss: 0.071746 Best loss: 0.071746 Accuracy: 98.08% 7 Validation loss: 0.069151 Best loss: 0.069151 Accuracy: 98.16% 8 Validation loss: 0.071440 Best loss: 0.069151 Accuracy: 98.40% 9 Validation loss: 0.073973 Best loss: 0.069151 Accuracy: 98.28% 10 Validation loss: 0.070183 Best loss: 0.069151 Accuracy: 98.32% 11 Validation loss: 0.074782 Best loss: 0.069151 Accuracy: 98.12% 12 Validation loss: 0.073066 Best loss: 0.069151 Accuracy: 98.12% 13 Validation loss: 0.064053 Best loss: 0.064053 Accuracy: 98.32% 14 Validation loss: 0.061407 Best loss: 0.061407 Accuracy: 98.40% 15 Validation loss: 0.069781 Best loss: 0.061407 Accuracy: 98.05% 16 Validation loss: 0.063385 Best loss: 0.061407 Accuracy: 98.48% 17 Validation loss: 0.066523 Best loss: 0.061407 Accuracy: 98.16% 18 Validation loss: 0.060093 Best loss: 0.060093 Accuracy: 98.48% 19 Validation loss: 0.060117 Best loss: 0.060093 Accuracy: 98.51% 20 Validation loss: 0.068550 Best loss: 0.060093 Accuracy: 98.40% 21 Validation loss: 0.058940 Best loss: 0.058940 Accuracy: 98.32% 22 Validation loss: 0.059043 Best loss: 0.058940 Accuracy: 98.40% 23 Validation loss: 0.061393 Best loss: 0.058940 Accuracy: 98.16% 24 Validation loss: 0.068982 Best loss: 0.058940 Accuracy: 98.08% 25 Validation loss: 0.058126 Best loss: 0.058126 Accuracy: 98.40% 26 Validation loss: 0.057792 Best loss: 0.057792 Accuracy: 98.51% 27 Validation loss: 0.066659 Best loss: 0.057792 Accuracy: 98.20% 28 Validation loss: 0.064132 Best loss: 0.057792 Accuracy: 98.20% 29 Validation loss: 0.060538 Best loss: 0.057792 Accuracy: 98.55% 30 Validation loss: 0.061174 Best loss: 0.057792 Accuracy: 98.36% 31 Validation loss: 0.057340 Best loss: 0.057340 Accuracy: 98.59% 32 Validation loss: 0.056163 Best loss: 0.056163 Accuracy: 98.63% 33 Validation loss: 0.057110 Best loss: 0.056163 Accuracy: 98.55% 34 Validation loss: 0.057037 Best loss: 0.056163 Accuracy: 98.59% 35 Validation loss: 0.055657 Best loss: 0.055657 Accuracy: 98.59% 36 Validation loss: 0.058836 Best loss: 0.055657 Accuracy: 98.48% 37 Validation loss: 0.056765 Best loss: 0.055657 Accuracy: 98.44% 38 Validation loss: 0.058205 Best loss: 0.055657 Accuracy: 98.55% 39 Validation loss: 0.058276 Best loss: 0.055657 Accuracy: 98.51% 40 Validation loss: 0.054396 Best loss: 0.054396 Accuracy: 98.55% 41 Validation loss: 0.062082 Best loss: 0.054396 Accuracy: 98.51% 42 Validation loss: 0.054962 Best loss: 0.054396 Accuracy: 98.51% 43 Validation loss: 0.057782 Best loss: 0.054396 Accuracy: 98.48% 44 Validation loss: 0.057868 Best loss: 0.054396 Accuracy: 98.59% 45 Validation loss: 0.059599 Best loss: 0.054396 Accuracy: 98.51% 46 Validation loss: 0.055743 Best loss: 0.054396 Accuracy: 98.48% 47 Validation loss: 0.060700 Best loss: 0.054396 Accuracy: 98.55% 48 Validation loss: 0.058618 Best loss: 0.054396 Accuracy: 98.59% 49 Validation loss: 0.062716 Best loss: 0.054396 Accuracy: 98.48% 50 Validation loss: 0.060097 Best loss: 0.054396 Accuracy: 98.51% 51 Validation loss: 0.061205 Best loss: 0.054396 Accuracy: 98.71% 52 Validation loss: 0.059605 Best loss: 0.054396 Accuracy: 98.32% 53 Validation loss: 0.060960 Best loss: 0.054396 Accuracy: 98.59% 54 Validation loss: 0.057499 Best loss: 0.054396 Accuracy: 98.44% 55 Validation loss: 0.061802 Best loss: 0.054396 Accuracy: 98.48% 56 Validation loss: 0.063113 Best loss: 0.054396 Accuracy: 98.55% 57 Validation loss: 0.055425 Best loss: 0.054396 Accuracy: 98.51% 58 Validation loss: 0.057835 Best loss: 0.054396 Accuracy: 98.59% 59 Validation loss: 0.066089 Best loss: 0.054396 Accuracy: 98.16% 60 Validation loss: 0.056930 Best loss: 0.054396 Accuracy: 98.55% 61 Validation loss: 0.062912 Best loss: 0.054396 Accuracy: 98.40% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=500, learning_rate=0.01, dropout_rate=0.4, total= 11.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 0.388304 Best loss: 0.388304 Accuracy: 90.66% 1 Validation loss: 0.237637 Best loss: 0.237637 Accuracy: 93.35% 2 Validation loss: 0.203182 Best loss: 0.203182 Accuracy: 94.88% 3 Validation loss: 0.186200 Best loss: 0.186200 Accuracy: 95.11% 4 Validation loss: 0.193086 Best loss: 0.186200 Accuracy: 94.57% 5 Validation loss: 0.199331 Best loss: 0.186200 Accuracy: 94.84% 6 Validation loss: 0.173756 Best loss: 0.173756 Accuracy: 95.07% 7 Validation loss: 0.172958 Best loss: 0.172958 Accuracy: 95.19% 8 Validation loss: 0.174749 Best loss: 0.172958 Accuracy: 94.68% 9 Validation loss: 0.168476 Best loss: 0.168476 Accuracy: 95.31% 10 Validation loss: 0.181074 Best loss: 0.168476 Accuracy: 95.74% 11 Validation loss: 0.217828 Best loss: 0.168476 Accuracy: 94.64% 12 Validation loss: 0.283667 Best loss: 0.168476 Accuracy: 94.84% 13 Validation loss: 0.172814 Best loss: 0.168476 Accuracy: 96.13% 14 Validation loss: 0.391857 Best loss: 0.168476 Accuracy: 86.40% 15 Validation loss: 0.409029 Best loss: 0.168476 Accuracy: 84.01% 16 Validation loss: 0.265770 Best loss: 0.168476 Accuracy: 93.04% 17 Validation loss: 0.230696 Best loss: 0.168476 Accuracy: 94.45% 18 Validation loss: 0.220363 Best loss: 0.168476 Accuracy: 93.98% 19 Validation loss: 0.287217 Best loss: 0.168476 Accuracy: 93.04% 20 Validation loss: 0.221689 Best loss: 0.168476 Accuracy: 94.37% 21 Validation loss: 0.249716 Best loss: 0.168476 Accuracy: 94.57% 22 Validation loss: 0.229006 Best loss: 0.168476 Accuracy: 94.25% 23 Validation loss: 0.193700 Best loss: 0.168476 Accuracy: 95.11% 24 Validation loss: 0.216743 Best loss: 0.168476 Accuracy: 94.61% 25 Validation loss: 0.199473 Best loss: 0.168476 Accuracy: 95.31% 26 Validation loss: 0.169888 Best loss: 0.168476 Accuracy: 95.70% 27 Validation loss: 0.173811 Best loss: 0.168476 Accuracy: 96.09% 28 Validation loss: 0.171152 Best loss: 0.168476 Accuracy: 95.86% 29 Validation loss: 0.158617 Best loss: 0.158617 Accuracy: 95.62% 30 Validation loss: 0.149299 Best loss: 0.149299 Accuracy: 96.09% 31 Validation loss: 0.176940 Best loss: 0.149299 Accuracy: 95.97% 32 Validation loss: 0.346764 Best loss: 0.149299 Accuracy: 89.37% 33 Validation loss: 1.657889 Best loss: 0.149299 Accuracy: 12.28% 34 Validation loss: 1.600897 Best loss: 0.149299 Accuracy: 25.33% 35 Validation loss: 1.536950 Best loss: 0.149299 Accuracy: 32.21% 36 Validation loss: 1.502955 Best loss: 0.149299 Accuracy: 30.10% 37 Validation loss: 1.077124 Best loss: 0.149299 Accuracy: 56.18% 38 Validation loss: 1.030134 Best loss: 0.149299 Accuracy: 50.66% 39 Validation loss: 1.028566 Best loss: 0.149299 Accuracy: 53.87% 40 Validation loss: 0.870308 Best loss: 0.149299 Accuracy: 59.54% 41 Validation loss: 0.846182 Best loss: 0.149299 Accuracy: 56.41% 42 Validation loss: 0.793852 Best loss: 0.149299 Accuracy: 60.44% 43 Validation loss: 0.805736 Best loss: 0.149299 Accuracy: 57.31% 44 Validation loss: 0.843530 Best loss: 0.149299 Accuracy: 58.56% 45 Validation loss: 0.797912 Best loss: 0.149299 Accuracy: 61.45% 46 Validation loss: 0.756201 Best loss: 0.149299 Accuracy: 60.01% 47 Validation loss: 0.717047 Best loss: 0.149299 Accuracy: 72.75% 48 Validation loss: 0.671060 Best loss: 0.149299 Accuracy: 71.93% 49 Validation loss: 0.571093 Best loss: 0.149299 Accuracy: 79.05% 50 Validation loss: 0.419975 Best loss: 0.149299 Accuracy: 83.19% 51 Validation loss: 0.735610 Best loss: 0.149299 Accuracy: 70.84% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2, total= 9.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 0.283520 Best loss: 0.283520 Accuracy: 92.26% 1 Validation loss: 0.212469 Best loss: 0.212469 Accuracy: 94.06% 2 Validation loss: 0.184947 Best loss: 0.184947 Accuracy: 94.92% 3 Validation loss: 0.168617 Best loss: 0.168617 Accuracy: 95.11% 4 Validation loss: 0.152311 Best loss: 0.152311 Accuracy: 95.43% 5 Validation loss: 0.139030 Best loss: 0.139030 Accuracy: 95.78% 6 Validation loss: 0.141574 Best loss: 0.139030 Accuracy: 96.05% 7 Validation loss: 0.141940 Best loss: 0.139030 Accuracy: 95.31% 8 Validation loss: 0.131972 Best loss: 0.131972 Accuracy: 95.50% 9 Validation loss: 0.131617 Best loss: 0.131617 Accuracy: 95.82% 10 Validation loss: 0.308651 Best loss: 0.131617 Accuracy: 90.30% 11 Validation loss: 0.607561 Best loss: 0.131617 Accuracy: 74.12% 12 Validation loss: 0.615766 Best loss: 0.131617 Accuracy: 74.59% 13 Validation loss: 0.449685 Best loss: 0.131617 Accuracy: 83.50% 14 Validation loss: 0.390241 Best loss: 0.131617 Accuracy: 87.26% 15 Validation loss: 0.405899 Best loss: 0.131617 Accuracy: 87.45% 16 Validation loss: 0.353587 Best loss: 0.131617 Accuracy: 89.41% 17 Validation loss: 0.318757 Best loss: 0.131617 Accuracy: 90.77% 18 Validation loss: 0.272331 Best loss: 0.131617 Accuracy: 92.38% 19 Validation loss: 0.244947 Best loss: 0.131617 Accuracy: 94.06% 20 Validation loss: 0.251289 Best loss: 0.131617 Accuracy: 92.77% 21 Validation loss: 0.219720 Best loss: 0.131617 Accuracy: 94.88% 22 Validation loss: 0.212530 Best loss: 0.131617 Accuracy: 95.35% 23 Validation loss: 0.215131 Best loss: 0.131617 Accuracy: 93.75% 24 Validation loss: 0.279818 Best loss: 0.131617 Accuracy: 92.96% 25 Validation loss: 0.265490 Best loss: 0.131617 Accuracy: 93.04% 26 Validation loss: 0.293174 Best loss: 0.131617 Accuracy: 91.48% 27 Validation loss: 0.253408 Best loss: 0.131617 Accuracy: 94.88% 28 Validation loss: 0.237296 Best loss: 0.131617 Accuracy: 93.35% 29 Validation loss: 0.248611 Best loss: 0.131617 Accuracy: 92.85% 30 Validation loss: 0.284230 Best loss: 0.131617 Accuracy: 93.63% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2, total= 6.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 0.261867 Best loss: 0.261867 Accuracy: 93.75% 1 Validation loss: 0.226739 Best loss: 0.226739 Accuracy: 93.47% 2 Validation loss: 0.173765 Best loss: 0.173765 Accuracy: 95.50% 3 Validation loss: 0.164019 Best loss: 0.164019 Accuracy: 95.78% 4 Validation loss: 0.165809 Best loss: 0.164019 Accuracy: 95.54% 5 Validation loss: 0.155774 Best loss: 0.155774 Accuracy: 96.36% 6 Validation loss: 0.177912 Best loss: 0.155774 Accuracy: 95.74% 7 Validation loss: 0.187432 Best loss: 0.155774 Accuracy: 95.93% 8 Validation loss: 0.154239 Best loss: 0.154239 Accuracy: 96.33% 9 Validation loss: 0.279446 Best loss: 0.154239 Accuracy: 93.90% 10 Validation loss: 0.848544 Best loss: 0.154239 Accuracy: 67.20% 11 Validation loss: 0.927900 Best loss: 0.154239 Accuracy: 55.55% 12 Validation loss: 1.148654 Best loss: 0.154239 Accuracy: 54.61% 13 Validation loss: 1.073501 Best loss: 0.154239 Accuracy: 58.44% 14 Validation loss: 0.913736 Best loss: 0.154239 Accuracy: 61.77% 15 Validation loss: 0.834042 Best loss: 0.154239 Accuracy: 57.62% 16 Validation loss: 0.769728 Best loss: 0.154239 Accuracy: 67.63% 17 Validation loss: 0.631751 Best loss: 0.154239 Accuracy: 70.41% 18 Validation loss: 0.464992 Best loss: 0.154239 Accuracy: 78.07% 19 Validation loss: 0.492176 Best loss: 0.154239 Accuracy: 80.53% 20 Validation loss: 0.457375 Best loss: 0.154239 Accuracy: 78.30% 21 Validation loss: 2.913650 Best loss: 0.154239 Accuracy: 37.41% 22 Validation loss: 1.304633 Best loss: 0.154239 Accuracy: 46.83% 23 Validation loss: 1.243046 Best loss: 0.154239 Accuracy: 45.86% 24 Validation loss: 1.151599 Best loss: 0.154239 Accuracy: 52.11% 25 Validation loss: 1.173820 Best loss: 0.154239 Accuracy: 43.55% 26 Validation loss: 1.108420 Best loss: 0.154239 Accuracy: 45.82% 27 Validation loss: 1.062640 Best loss: 0.154239 Accuracy: 48.75% 28 Validation loss: 0.998246 Best loss: 0.154239 Accuracy: 51.56% 29 Validation loss: 0.870770 Best loss: 0.154239 Accuracy: 57.19% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=10, batch_size=500, learning_rate=0.1, dropout_rate=0.2, total= 6.2s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.100162 Best loss: 0.100162 Accuracy: 97.58% 1 Validation loss: 0.102482 Best loss: 0.100162 Accuracy: 97.77% 2 Validation loss: 0.092138 Best loss: 0.092138 Accuracy: 97.38% 3 Validation loss: 0.078162 Best loss: 0.078162 Accuracy: 98.08% 4 Validation loss: 0.092880 Best loss: 0.078162 Accuracy: 97.65% 5 Validation loss: 0.167062 Best loss: 0.078162 Accuracy: 97.65% 6 Validation loss: 0.106566 Best loss: 0.078162 Accuracy: 98.20% 7 Validation loss: 0.066188 Best loss: 0.066188 Accuracy: 98.32% 8 Validation loss: 0.052193 Best loss: 0.052193 Accuracy: 98.40% 9 Validation loss: 0.069810 Best loss: 0.052193 Accuracy: 98.32% 10 Validation loss: 0.076071 Best loss: 0.052193 Accuracy: 98.24% 11 Validation loss: 0.067161 Best loss: 0.052193 Accuracy: 98.20% 12 Validation loss: 0.087684 Best loss: 0.052193 Accuracy: 98.32% 13 Validation loss: 0.103299 Best loss: 0.052193 Accuracy: 97.62% 14 Validation loss: 0.105147 Best loss: 0.052193 Accuracy: 98.20% 15 Validation loss: 0.068423 Best loss: 0.052193 Accuracy: 98.28% 16 Validation loss: 0.062607 Best loss: 0.052193 Accuracy: 98.63% 17 Validation loss: 0.059118 Best loss: 0.052193 Accuracy: 98.51% 18 Validation loss: 0.163362 Best loss: 0.052193 Accuracy: 98.44% 19 Validation loss: 0.111824 Best loss: 0.052193 Accuracy: 98.79% 20 Validation loss: 0.068144 Best loss: 0.052193 Accuracy: 98.32% 21 Validation loss: 0.523986 Best loss: 0.052193 Accuracy: 94.14% 22 Validation loss: 0.534044 Best loss: 0.052193 Accuracy: 96.36% 23 Validation loss: 0.176717 Best loss: 0.052193 Accuracy: 96.29% 24 Validation loss: 0.435227 Best loss: 0.052193 Accuracy: 85.14% 25 Validation loss: 0.255200 Best loss: 0.052193 Accuracy: 93.55% 26 Validation loss: 0.388393 Best loss: 0.052193 Accuracy: 91.32% 27 Validation loss: 0.493127 Best loss: 0.052193 Accuracy: 82.76% 28 Validation loss: 0.372801 Best loss: 0.052193 Accuracy: 92.96% 29 Validation loss: 0.544141 Best loss: 0.052193 Accuracy: 93.39% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 18.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.149001 Best loss: 0.149001 Accuracy: 96.79% 1 Validation loss: 0.099493 Best loss: 0.099493 Accuracy: 97.42% 2 Validation loss: 0.088264 Best loss: 0.088264 Accuracy: 97.50% 3 Validation loss: 0.114833 Best loss: 0.088264 Accuracy: 97.85% 4 Validation loss: 0.100439 Best loss: 0.088264 Accuracy: 97.89% 5 Validation loss: 0.099403 Best loss: 0.088264 Accuracy: 97.65% 6 Validation loss: 0.081741 Best loss: 0.081741 Accuracy: 98.36% 7 Validation loss: 0.080358 Best loss: 0.080358 Accuracy: 98.40% 8 Validation loss: 0.084608 Best loss: 0.080358 Accuracy: 98.12% 9 Validation loss: 0.063037 Best loss: 0.063037 Accuracy: 98.36% 10 Validation loss: 0.061860 Best loss: 0.061860 Accuracy: 98.48% 11 Validation loss: 0.083204 Best loss: 0.061860 Accuracy: 98.24% 12 Validation loss: 0.084315 Best loss: 0.061860 Accuracy: 98.48% 13 Validation loss: 0.059030 Best loss: 0.059030 Accuracy: 98.55% 14 Validation loss: 0.088164 Best loss: 0.059030 Accuracy: 97.85% 15 Validation loss: 0.098789 Best loss: 0.059030 Accuracy: 97.93% 16 Validation loss: 0.089897 Best loss: 0.059030 Accuracy: 98.08% 17 Validation loss: 0.074710 Best loss: 0.059030 Accuracy: 98.55% 18 Validation loss: 0.082832 Best loss: 0.059030 Accuracy: 98.67% 19 Validation loss: 0.059001 Best loss: 0.059001 Accuracy: 98.75% 20 Validation loss: 0.080305 Best loss: 0.059001 Accuracy: 98.59% 21 Validation loss: 0.078739 Best loss: 0.059001 Accuracy: 98.51% 22 Validation loss: 0.077338 Best loss: 0.059001 Accuracy: 98.32% 23 Validation loss: 0.067237 Best loss: 0.059001 Accuracy: 98.40% 24 Validation loss: 0.058196 Best loss: 0.058196 Accuracy: 98.36% 25 Validation loss: 0.063650 Best loss: 0.058196 Accuracy: 98.63% 26 Validation loss: 0.065839 Best loss: 0.058196 Accuracy: 98.59% 27 Validation loss: 0.056168 Best loss: 0.056168 Accuracy: 98.71% 28 Validation loss: 0.074834 Best loss: 0.056168 Accuracy: 98.67% 29 Validation loss: 0.068290 Best loss: 0.056168 Accuracy: 98.48% 30 Validation loss: 0.091986 Best loss: 0.056168 Accuracy: 98.32% 31 Validation loss: 0.063159 Best loss: 0.056168 Accuracy: 98.55% 32 Validation loss: 0.056304 Best loss: 0.056168 Accuracy: 98.55% 33 Validation loss: 0.054261 Best loss: 0.054261 Accuracy: 98.63% 34 Validation loss: 0.058819 Best loss: 0.054261 Accuracy: 98.67% 35 Validation loss: 0.065078 Best loss: 0.054261 Accuracy: 98.48% 36 Validation loss: 0.087454 Best loss: 0.054261 Accuracy: 98.28% 37 Validation loss: 0.079882 Best loss: 0.054261 Accuracy: 98.48% 38 Validation loss: 0.119406 Best loss: 0.054261 Accuracy: 98.44% 39 Validation loss: 0.088473 Best loss: 0.054261 Accuracy: 98.36% 40 Validation loss: 0.079727 Best loss: 0.054261 Accuracy: 98.51% 41 Validation loss: 0.085875 Best loss: 0.054261 Accuracy: 98.71% 42 Validation loss: 0.115107 Best loss: 0.054261 Accuracy: 98.01% 43 Validation loss: 0.083212 Best loss: 0.054261 Accuracy: 98.44% 44 Validation loss: 0.072160 Best loss: 0.054261 Accuracy: 98.44% 45 Validation loss: 0.089514 Best loss: 0.054261 Accuracy: 98.16% 46 Validation loss: 0.192859 Best loss: 0.054261 Accuracy: 98.71% 47 Validation loss: 0.164841 Best loss: 0.054261 Accuracy: 98.48% 48 Validation loss: 0.420732 Best loss: 0.054261 Accuracy: 98.44% 49 Validation loss: 0.108097 Best loss: 0.054261 Accuracy: 98.12% 50 Validation loss: 0.092822 Best loss: 0.054261 Accuracy: 98.28% 51 Validation loss: 0.116728 Best loss: 0.054261 Accuracy: 98.36% 52 Validation loss: 0.063501 Best loss: 0.054261 Accuracy: 98.51% 53 Validation loss: 0.057549 Best loss: 0.054261 Accuracy: 98.44% 54 Validation loss: 0.081409 Best loss: 0.054261 Accuracy: 98.63% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 32.3s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.097507 Best loss: 0.097507 Accuracy: 97.54% 1 Validation loss: 0.082819 Best loss: 0.082819 Accuracy: 98.24% 2 Validation loss: 0.096354 Best loss: 0.082819 Accuracy: 97.58% 3 Validation loss: 0.068670 Best loss: 0.068670 Accuracy: 98.28% 4 Validation loss: 0.083069 Best loss: 0.068670 Accuracy: 98.28% 5 Validation loss: 0.096050 Best loss: 0.068670 Accuracy: 98.48% 6 Validation loss: 0.070906 Best loss: 0.068670 Accuracy: 98.20% 7 Validation loss: 0.114107 Best loss: 0.068670 Accuracy: 98.05% 8 Validation loss: 0.101567 Best loss: 0.068670 Accuracy: 98.05% 9 Validation loss: 0.064929 Best loss: 0.064929 Accuracy: 98.83% 10 Validation loss: 0.088059 Best loss: 0.064929 Accuracy: 98.48% 11 Validation loss: 0.096119 Best loss: 0.064929 Accuracy: 98.32% 12 Validation loss: 0.071822 Best loss: 0.064929 Accuracy: 98.48% 13 Validation loss: 0.078048 Best loss: 0.064929 Accuracy: 98.71% 14 Validation loss: 0.081716 Best loss: 0.064929 Accuracy: 98.63% 15 Validation loss: 0.084301 Best loss: 0.064929 Accuracy: 98.67% 16 Validation loss: 0.099360 Best loss: 0.064929 Accuracy: 98.05% 17 Validation loss: 0.179076 Best loss: 0.064929 Accuracy: 98.36% 18 Validation loss: 0.100261 Best loss: 0.064929 Accuracy: 98.32% 19 Validation loss: 0.156866 Best loss: 0.064929 Accuracy: 97.50% 20 Validation loss: 0.246759 Best loss: 0.064929 Accuracy: 97.85% 21 Validation loss: 0.125512 Best loss: 0.064929 Accuracy: 97.93% 22 Validation loss: 0.217282 Best loss: 0.064929 Accuracy: 98.01% 23 Validation loss: 0.133932 Best loss: 0.064929 Accuracy: 97.26% 24 Validation loss: 0.160062 Best loss: 0.064929 Accuracy: 97.89% 25 Validation loss: 0.167562 Best loss: 0.064929 Accuracy: 98.28% 26 Validation loss: 0.120959 Best loss: 0.064929 Accuracy: 98.36% 27 Validation loss: 0.123431 Best loss: 0.064929 Accuracy: 98.63% 28 Validation loss: 0.135786 Best loss: 0.064929 Accuracy: 98.63% 29 Validation loss: 0.144110 Best loss: 0.064929 Accuracy: 98.59% 30 Validation loss: 0.093766 Best loss: 0.064929 Accuracy: 98.08% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 18.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1341.429199 Best loss: 1341.429199 Accuracy: 19.19% 1 Validation loss: 26982.443359 Best loss: 1341.429199 Accuracy: 18.73% 2 Validation loss: 4562.629395 Best loss: 1341.429199 Accuracy: 19.27% 3 Validation loss: 21713.029297 Best loss: 1341.429199 Accuracy: 18.73% 4 Validation loss: 9832.622070 Best loss: 1341.429199 Accuracy: 15.95% 5 Validation loss: 15241.589844 Best loss: 1341.429199 Accuracy: 18.88% 6 Validation loss: 156215.203125 Best loss: 1341.429199 Accuracy: 19.27% 7 Validation loss: 401745.656250 Best loss: 1341.429199 Accuracy: 20.91% 8 Validation loss: 4554170.500000 Best loss: 1341.429199 Accuracy: 19.08% 9 Validation loss: 225429.812500 Best loss: 1341.429199 Accuracy: 19.27% 10 Validation loss: 122199.859375 Best loss: 1341.429199 Accuracy: 19.08% 11 Validation loss: 196524.031250 Best loss: 1341.429199 Accuracy: 22.01% 12 Validation loss: 143785.062500 Best loss: 1341.429199 Accuracy: 19.27% 13 Validation loss: 1032626.062500 Best loss: 1341.429199 Accuracy: 18.61% 14 Validation loss: 105960.445312 Best loss: 1341.429199 Accuracy: 18.73% 15 Validation loss: 88687.218750 Best loss: 1341.429199 Accuracy: 18.73% 16 Validation loss: 863240.687500 Best loss: 1341.429199 Accuracy: 19.08% 17 Validation loss: 298156.437500 Best loss: 1341.429199 Accuracy: 22.01% 18 Validation loss: 157287.062500 Best loss: 1341.429199 Accuracy: 19.27% 19 Validation loss: 238239.781250 Best loss: 1341.429199 Accuracy: 18.73% 20 Validation loss: 133207.234375 Best loss: 1341.429199 Accuracy: 19.08% 21 Validation loss: 241600.250000 Best loss: 1341.429199 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5, total= 28.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 89104.492188 Best loss: 89104.492188 Accuracy: 19.08% 1 Validation loss: 25270.927734 Best loss: 25270.927734 Accuracy: 19.08% 2 Validation loss: 23797.966797 Best loss: 23797.966797 Accuracy: 22.01% 3 Validation loss: 23407.605469 Best loss: 23407.605469 Accuracy: 18.73% 4 Validation loss: 70701.843750 Best loss: 23407.605469 Accuracy: 18.73% 5 Validation loss: 71244.218750 Best loss: 23407.605469 Accuracy: 20.91% 6 Validation loss: 686272.687500 Best loss: 23407.605469 Accuracy: 21.89% 7 Validation loss: 150318.625000 Best loss: 23407.605469 Accuracy: 18.73% 8 Validation loss: 57252.753906 Best loss: 23407.605469 Accuracy: 22.01% 9 Validation loss: 31728.466797 Best loss: 23407.605469 Accuracy: 19.27% 10 Validation loss: 85564.406250 Best loss: 23407.605469 Accuracy: 24.63% 11 Validation loss: 76173.710938 Best loss: 23407.605469 Accuracy: 18.73% 12 Validation loss: 60389.781250 Best loss: 23407.605469 Accuracy: 20.91% 13 Validation loss: 74304.890625 Best loss: 23407.605469 Accuracy: 22.01% 14 Validation loss: 81314.898438 Best loss: 23407.605469 Accuracy: 18.73% 15 Validation loss: 51023.839844 Best loss: 23407.605469 Accuracy: 35.03% 16 Validation loss: 29149.529297 Best loss: 23407.605469 Accuracy: 18.73% 17 Validation loss: 29666.552734 Best loss: 23407.605469 Accuracy: 20.91% 18 Validation loss: 22883.916016 Best loss: 22883.916016 Accuracy: 18.73% 19 Validation loss: 10605.161133 Best loss: 10605.161133 Accuracy: 22.01% 20 Validation loss: 22509.085938 Best loss: 10605.161133 Accuracy: 18.73% 21 Validation loss: 11738.761719 Best loss: 10605.161133 Accuracy: 22.01% 22 Validation loss: 25821.000000 Best loss: 10605.161133 Accuracy: 18.73% 23 Validation loss: 44073.128906 Best loss: 10605.161133 Accuracy: 22.01% 24 Validation loss: 42658.039062 Best loss: 10605.161133 Accuracy: 18.73% 25 Validation loss: 23657.347656 Best loss: 10605.161133 Accuracy: 19.08% 26 Validation loss: 12439.087891 Best loss: 10605.161133 Accuracy: 19.08% 27 Validation loss: 42334.621094 Best loss: 10605.161133 Accuracy: 18.73% 28 Validation loss: 18338.019531 Best loss: 10605.161133 Accuracy: 22.01% 29 Validation loss: 7649.437012 Best loss: 7649.437012 Accuracy: 19.08% 30 Validation loss: 10920.633789 Best loss: 7649.437012 Accuracy: 18.73% 31 Validation loss: 16131.978516 Best loss: 7649.437012 Accuracy: 18.73% 32 Validation loss: 27866.914062 Best loss: 7649.437012 Accuracy: 18.73% 33 Validation loss: 24726.179688 Best loss: 7649.437012 Accuracy: 18.73% 34 Validation loss: 66514.359375 Best loss: 7649.437012 Accuracy: 20.91% 35 Validation loss: 12691.410156 Best loss: 7649.437012 Accuracy: 20.91% 36 Validation loss: 13465.809570 Best loss: 7649.437012 Accuracy: 18.73% 37 Validation loss: 21827.640625 Best loss: 7649.437012 Accuracy: 22.01% 38 Validation loss: 26196.617188 Best loss: 7649.437012 Accuracy: 18.73% 39 Validation loss: 42957.593750 Best loss: 7649.437012 Accuracy: 19.27% 40 Validation loss: 29906.892578 Best loss: 7649.437012 Accuracy: 20.91% 41 Validation loss: 2154.759521 Best loss: 2154.759521 Accuracy: 31.59% 42 Validation loss: 49008.535156 Best loss: 2154.759521 Accuracy: 18.73% 43 Validation loss: 7171.108887 Best loss: 2154.759521 Accuracy: 18.73% 44 Validation loss: 12317.237305 Best loss: 2154.759521 Accuracy: 22.01% 45 Validation loss: 6403.500977 Best loss: 2154.759521 Accuracy: 19.78% 46 Validation loss: 1053308.750000 Best loss: 2154.759521 Accuracy: 20.91% 47 Validation loss: 254726.500000 Best loss: 2154.759521 Accuracy: 19.27% 48 Validation loss: 87023.304688 Best loss: 2154.759521 Accuracy: 19.27% 49 Validation loss: 428890.781250 Best loss: 2154.759521 Accuracy: 19.27% 50 Validation loss: 134302.656250 Best loss: 2154.759521 Accuracy: 22.01% 51 Validation loss: 69275.515625 Best loss: 2154.759521 Accuracy: 22.01% 52 Validation loss: 141941.468750 Best loss: 2154.759521 Accuracy: 18.73% 53 Validation loss: 372458.437500 Best loss: 2154.759521 Accuracy: 22.01% 54 Validation loss: 251079.359375 Best loss: 2154.759521 Accuracy: 22.01% 55 Validation loss: 134051.093750 Best loss: 2154.759521 Accuracy: 20.91% 56 Validation loss: 310615.406250 Best loss: 2154.759521 Accuracy: 19.08% 57 Validation loss: 69752.531250 Best loss: 2154.759521 Accuracy: 18.73% 58 Validation loss: 86794.632812 Best loss: 2154.759521 Accuracy: 17.71% 59 Validation loss: 116716.906250 Best loss: 2154.759521 Accuracy: 18.73% 60 Validation loss: 390809.812500 Best loss: 2154.759521 Accuracy: 18.73% 61 Validation loss: 197522.796875 Best loss: 2154.759521 Accuracy: 20.91% 62 Validation loss: 792859.250000 Best loss: 2154.759521 Accuracy: 19.27% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5, total= 1.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1217.733887 Best loss: 1217.733887 Accuracy: 19.27% 1 Validation loss: 324979.375000 Best loss: 1217.733887 Accuracy: 19.27% 2 Validation loss: 285625.531250 Best loss: 1217.733887 Accuracy: 20.91% 3 Validation loss: 46551.589844 Best loss: 1217.733887 Accuracy: 19.08% 4 Validation loss: 105524.867188 Best loss: 1217.733887 Accuracy: 19.27% 5 Validation loss: 41528.167969 Best loss: 1217.733887 Accuracy: 19.08% 6 Validation loss: 42364.140625 Best loss: 1217.733887 Accuracy: 18.73% 7 Validation loss: 36540.195312 Best loss: 1217.733887 Accuracy: 18.73% 8 Validation loss: 16461.572266 Best loss: 1217.733887 Accuracy: 18.96% 9 Validation loss: 14257.327148 Best loss: 1217.733887 Accuracy: 14.19% 10 Validation loss: 27536.296875 Best loss: 1217.733887 Accuracy: 20.91% 11 Validation loss: 11319.768555 Best loss: 1217.733887 Accuracy: 19.70% 12 Validation loss: 93388.312500 Best loss: 1217.733887 Accuracy: 22.01% 13 Validation loss: 27505.439453 Best loss: 1217.733887 Accuracy: 19.27% 14 Validation loss: 9092.140625 Best loss: 1217.733887 Accuracy: 18.73% 15 Validation loss: 29568.052734 Best loss: 1217.733887 Accuracy: 20.91% 16 Validation loss: 129083.007812 Best loss: 1217.733887 Accuracy: 19.08% 17 Validation loss: 248848.109375 Best loss: 1217.733887 Accuracy: 20.91% 18 Validation loss: 69768.890625 Best loss: 1217.733887 Accuracy: 22.01% 19 Validation loss: 1441604.500000 Best loss: 1217.733887 Accuracy: 19.08% 20 Validation loss: 322958.750000 Best loss: 1217.733887 Accuracy: 19.27% 21 Validation loss: 79191.484375 Best loss: 1217.733887 Accuracy: 18.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.5, total= 28.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 2.777014 Best loss: 2.777014 Accuracy: 19.27% 1 Validation loss: 2.493246 Best loss: 2.493246 Accuracy: 18.73% 2 Validation loss: 2.281677 Best loss: 2.281677 Accuracy: 19.27% 3 Validation loss: 3.042454 Best loss: 2.281677 Accuracy: 22.01% 4 Validation loss: 2.718512 Best loss: 2.281677 Accuracy: 19.27% 5 Validation loss: 2.292811 Best loss: 2.281677 Accuracy: 20.91% 6 Validation loss: 2.049773 Best loss: 2.049773 Accuracy: 18.73% 7 Validation loss: 1.909894 Best loss: 1.909894 Accuracy: 20.91% 8 Validation loss: 2.701862 Best loss: 1.909894 Accuracy: 19.08% 9 Validation loss: 2.218164 Best loss: 1.909894 Accuracy: 19.27% 10 Validation loss: 1.903213 Best loss: 1.903213 Accuracy: 19.27% 11 Validation loss: 2.101270 Best loss: 1.903213 Accuracy: 19.08% 12 Validation loss: 2.090753 Best loss: 1.903213 Accuracy: 22.01% 13 Validation loss: 2.276820 Best loss: 1.903213 Accuracy: 22.01% 14 Validation loss: 3.238749 Best loss: 1.903213 Accuracy: 18.73% 15 Validation loss: 2.468314 Best loss: 1.903213 Accuracy: 19.08% 16 Validation loss: 2.552812 Best loss: 1.903213 Accuracy: 18.73% 17 Validation loss: 2.279547 Best loss: 1.903213 Accuracy: 20.91% 18 Validation loss: 3.453777 Best loss: 1.903213 Accuracy: 22.01% 19 Validation loss: 2.956371 Best loss: 1.903213 Accuracy: 22.01% 20 Validation loss: 3.392728 Best loss: 1.903213 Accuracy: 20.91% 21 Validation loss: 2.013876 Best loss: 1.903213 Accuracy: 20.91% 22 Validation loss: 2.057791 Best loss: 1.903213 Accuracy: 18.73% 23 Validation loss: 2.115132 Best loss: 1.903213 Accuracy: 19.08% 24 Validation loss: 2.480372 Best loss: 1.903213 Accuracy: 20.91% 25 Validation loss: 2.587861 Best loss: 1.903213 Accuracy: 20.91% 26 Validation loss: 2.666172 Best loss: 1.903213 Accuracy: 20.91% 27 Validation loss: 1.842633 Best loss: 1.842633 Accuracy: 19.08% 28 Validation loss: 2.438254 Best loss: 1.842633 Accuracy: 19.08% 29 Validation loss: 2.189014 Best loss: 1.842633 Accuracy: 19.27% 30 Validation loss: 1.703390 Best loss: 1.703390 Accuracy: 20.91% 31 Validation loss: 2.334586 Best loss: 1.703390 Accuracy: 19.08% 32 Validation loss: 2.185187 Best loss: 1.703390 Accuracy: 20.91% 33 Validation loss: 1.719668 Best loss: 1.703390 Accuracy: 22.01% 34 Validation loss: 1.628595 Best loss: 1.628595 Accuracy: 22.01% 35 Validation loss: 3.032679 Best loss: 1.628595 Accuracy: 19.27% 36 Validation loss: 2.421143 Best loss: 1.628595 Accuracy: 22.01% 37 Validation loss: 2.550120 Best loss: 1.628595 Accuracy: 20.91% 38 Validation loss: 1.910114 Best loss: 1.628595 Accuracy: 22.01% 39 Validation loss: 2.513147 Best loss: 1.628595 Accuracy: 18.73% 40 Validation loss: 2.453553 Best loss: 1.628595 Accuracy: 22.01% 41 Validation loss: 2.396963 Best loss: 1.628595 Accuracy: 19.27% 42 Validation loss: 2.810289 Best loss: 1.628595 Accuracy: 22.01% 43 Validation loss: 2.731160 Best loss: 1.628595 Accuracy: 18.73% 44 Validation loss: 3.093425 Best loss: 1.628595 Accuracy: 20.91% 45 Validation loss: 2.365693 Best loss: 1.628595 Accuracy: 18.73% 46 Validation loss: 2.828111 Best loss: 1.628595 Accuracy: 18.73% 47 Validation loss: 2.716028 Best loss: 1.628595 Accuracy: 19.08% 48 Validation loss: 2.009304 Best loss: 1.628595 Accuracy: 19.08% 49 Validation loss: 2.157573 Best loss: 1.628595 Accuracy: 19.08% 50 Validation loss: 1.807541 Best loss: 1.628595 Accuracy: 20.91% 51 Validation loss: 2.615746 Best loss: 1.628595 Accuracy: 22.01% 52 Validation loss: 1.928062 Best loss: 1.628595 Accuracy: 19.08% 53 Validation loss: 3.122555 Best loss: 1.628595 Accuracy: 19.27% 54 Validation loss: 2.647550 Best loss: 1.628595 Accuracy: 22.01% 55 Validation loss: 2.229312 Best loss: 1.628595 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 4.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 2.212627 Best loss: 2.212627 Accuracy: 19.27% 1 Validation loss: 2.170536 Best loss: 2.170536 Accuracy: 19.27% 2 Validation loss: 1.816077 Best loss: 1.816077 Accuracy: 22.01% 3 Validation loss: 2.100055 Best loss: 1.816077 Accuracy: 22.01% 4 Validation loss: 1.901633 Best loss: 1.816077 Accuracy: 19.08% 5 Validation loss: 2.054681 Best loss: 1.816077 Accuracy: 18.73% 6 Validation loss: 2.198756 Best loss: 1.816077 Accuracy: 19.27% 7 Validation loss: 2.570518 Best loss: 1.816077 Accuracy: 20.91% 8 Validation loss: 2.631223 Best loss: 1.816077 Accuracy: 18.73% 9 Validation loss: 1.970993 Best loss: 1.816077 Accuracy: 22.01% 10 Validation loss: 2.443515 Best loss: 1.816077 Accuracy: 19.08% 11 Validation loss: 2.979535 Best loss: 1.816077 Accuracy: 19.08% 12 Validation loss: 2.325011 Best loss: 1.816077 Accuracy: 20.91% 13 Validation loss: 2.824593 Best loss: 1.816077 Accuracy: 19.08% 14 Validation loss: 2.958388 Best loss: 1.816077 Accuracy: 19.27% 15 Validation loss: 2.465580 Best loss: 1.816077 Accuracy: 20.91% 16 Validation loss: 1.683397 Best loss: 1.683397 Accuracy: 22.01% 17 Validation loss: 2.397759 Best loss: 1.683397 Accuracy: 18.73% 18 Validation loss: 2.232680 Best loss: 1.683397 Accuracy: 19.08% 19 Validation loss: 2.322164 Best loss: 1.683397 Accuracy: 18.73% 20 Validation loss: 1.726230 Best loss: 1.683397 Accuracy: 18.73% 21 Validation loss: 1.970972 Best loss: 1.683397 Accuracy: 22.01% 22 Validation loss: 1.679338 Best loss: 1.679338 Accuracy: 18.73% 23 Validation loss: 3.116611 Best loss: 1.679338 Accuracy: 19.27% 24 Validation loss: 1.897403 Best loss: 1.679338 Accuracy: 18.73% 25 Validation loss: 1.984912 Best loss: 1.679338 Accuracy: 22.01% 26 Validation loss: 3.022332 Best loss: 1.679338 Accuracy: 19.08% 27 Validation loss: 2.557768 Best loss: 1.679338 Accuracy: 20.91% 28 Validation loss: 2.303726 Best loss: 1.679338 Accuracy: 20.91% 29 Validation loss: 2.091059 Best loss: 1.679338 Accuracy: 19.08% 30 Validation loss: 2.283423 Best loss: 1.679338 Accuracy: 22.01% 31 Validation loss: 2.333198 Best loss: 1.679338 Accuracy: 19.08% 32 Validation loss: 2.237311 Best loss: 1.679338 Accuracy: 18.73% 33 Validation loss: 3.356252 Best loss: 1.679338 Accuracy: 19.08% 34 Validation loss: 1.959495 Best loss: 1.679338 Accuracy: 22.01% 35 Validation loss: 2.733108 Best loss: 1.679338 Accuracy: 22.01% 36 Validation loss: 3.177168 Best loss: 1.679338 Accuracy: 19.27% 37 Validation loss: 2.645809 Best loss: 1.679338 Accuracy: 20.91% 38 Validation loss: 1.747638 Best loss: 1.679338 Accuracy: 18.73% 39 Validation loss: 5.013640 Best loss: 1.679338 Accuracy: 18.73% 40 Validation loss: 2.815721 Best loss: 1.679338 Accuracy: 18.73% 41 Validation loss: 4.490565 Best loss: 1.679338 Accuracy: 19.27% 42 Validation loss: 2.966645 Best loss: 1.679338 Accuracy: 19.27% 43 Validation loss: 2.568960 Best loss: 1.679338 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 3.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4 0 Validation loss: 1.837926 Best loss: 1.837926 Accuracy: 22.01% 1 Validation loss: 2.305294 Best loss: 1.837926 Accuracy: 22.01% 2 Validation loss: 2.465991 Best loss: 1.837926 Accuracy: 19.08% 3 Validation loss: 2.738959 Best loss: 1.837926 Accuracy: 22.01% 4 Validation loss: 3.856620 Best loss: 1.837926 Accuracy: 20.91% 5 Validation loss: 2.399260 Best loss: 1.837926 Accuracy: 20.91% 6 Validation loss: 1.989864 Best loss: 1.837926 Accuracy: 22.01% 7 Validation loss: 1.672320 Best loss: 1.672320 Accuracy: 20.91% 8 Validation loss: 1.697073 Best loss: 1.672320 Accuracy: 22.01% 9 Validation loss: 2.448783 Best loss: 1.672320 Accuracy: 18.73% 10 Validation loss: 2.339123 Best loss: 1.672320 Accuracy: 18.73% 11 Validation loss: 2.186826 Best loss: 1.672320 Accuracy: 18.73% 12 Validation loss: 2.209591 Best loss: 1.672320 Accuracy: 22.01% 13 Validation loss: 3.057066 Best loss: 1.672320 Accuracy: 19.08% 14 Validation loss: 1.883864 Best loss: 1.672320 Accuracy: 22.01% 15 Validation loss: 2.201201 Best loss: 1.672320 Accuracy: 19.27% 16 Validation loss: 1.972194 Best loss: 1.672320 Accuracy: 19.08% 17 Validation loss: 2.398799 Best loss: 1.672320 Accuracy: 18.73% 18 Validation loss: 1.799597 Best loss: 1.672320 Accuracy: 20.91% 19 Validation loss: 3.239805 Best loss: 1.672320 Accuracy: 22.01% 20 Validation loss: 1.921057 Best loss: 1.672320 Accuracy: 20.91% 21 Validation loss: 3.416172 Best loss: 1.672320 Accuracy: 22.01% 22 Validation loss: 2.283767 Best loss: 1.672320 Accuracy: 20.91% 23 Validation loss: 3.748461 Best loss: 1.672320 Accuracy: 19.08% 24 Validation loss: 2.545792 Best loss: 1.672320 Accuracy: 19.27% 25 Validation loss: 1.988029 Best loss: 1.672320 Accuracy: 18.73% 26 Validation loss: 2.036219 Best loss: 1.672320 Accuracy: 18.73% 27 Validation loss: 2.330848 Best loss: 1.672320 Accuracy: 20.91% 28 Validation loss: 3.196505 Best loss: 1.672320 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.4, total= 2.5min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 2.156161 Best loss: 2.156161 Accuracy: 18.73% 1 Validation loss: 2.063816 Best loss: 2.063816 Accuracy: 19.08% 2 Validation loss: 2.359565 Best loss: 2.063816 Accuracy: 19.27% 3 Validation loss: 1.871980 Best loss: 1.871980 Accuracy: 20.91% 4 Validation loss: 2.768074 Best loss: 1.871980 Accuracy: 18.73% 5 Validation loss: 1.780415 Best loss: 1.780415 Accuracy: 22.01% 6 Validation loss: 2.011012 Best loss: 1.780415 Accuracy: 18.73% 7 Validation loss: 1.786950 Best loss: 1.780415 Accuracy: 20.91% 8 Validation loss: 2.677289 Best loss: 1.780415 Accuracy: 19.08% 9 Validation loss: 2.138432 Best loss: 1.780415 Accuracy: 20.91% 10 Validation loss: 1.957773 Best loss: 1.780415 Accuracy: 19.27% 11 Validation loss: 2.481015 Best loss: 1.780415 Accuracy: 19.08% 12 Validation loss: 2.067203 Best loss: 1.780415 Accuracy: 22.01% 13 Validation loss: 2.136309 Best loss: 1.780415 Accuracy: 22.01% 14 Validation loss: 3.276874 Best loss: 1.780415 Accuracy: 20.91% 15 Validation loss: 2.067671 Best loss: 1.780415 Accuracy: 19.27% 16 Validation loss: 2.272886 Best loss: 1.780415 Accuracy: 18.73% 17 Validation loss: 2.634235 Best loss: 1.780415 Accuracy: 20.91% 18 Validation loss: 3.481533 Best loss: 1.780415 Accuracy: 22.01% 19 Validation loss: 2.956635 Best loss: 1.780415 Accuracy: 22.01% 20 Validation loss: 3.560570 Best loss: 1.780415 Accuracy: 20.91% 21 Validation loss: 2.339725 Best loss: 1.780415 Accuracy: 20.91% 22 Validation loss: 2.140547 Best loss: 1.780415 Accuracy: 18.73% 23 Validation loss: 1.855719 Best loss: 1.780415 Accuracy: 19.08% 24 Validation loss: 2.405271 Best loss: 1.780415 Accuracy: 20.91% 25 Validation loss: 1.994892 Best loss: 1.780415 Accuracy: 19.27% 26 Validation loss: 2.450106 Best loss: 1.780415 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3, total= 2.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.989717 Best loss: 1.989717 Accuracy: 19.08% 1 Validation loss: 2.320880 Best loss: 1.989717 Accuracy: 19.27% 2 Validation loss: 2.109937 Best loss: 1.989717 Accuracy: 22.01% 3 Validation loss: 2.416858 Best loss: 1.989717 Accuracy: 19.08% 4 Validation loss: 1.856821 Best loss: 1.856821 Accuracy: 19.08% 5 Validation loss: 1.975856 Best loss: 1.856821 Accuracy: 18.73% 6 Validation loss: 2.763820 Best loss: 1.856821 Accuracy: 18.73% 7 Validation loss: 2.114563 Best loss: 1.856821 Accuracy: 19.08% 8 Validation loss: 2.744597 Best loss: 1.856821 Accuracy: 18.73% 9 Validation loss: 1.979291 Best loss: 1.856821 Accuracy: 22.01% 10 Validation loss: 2.352546 Best loss: 1.856821 Accuracy: 19.08% 11 Validation loss: 2.828781 Best loss: 1.856821 Accuracy: 19.08% 12 Validation loss: 2.340877 Best loss: 1.856821 Accuracy: 20.91% 13 Validation loss: 3.272311 Best loss: 1.856821 Accuracy: 19.27% 14 Validation loss: 2.961215 Best loss: 1.856821 Accuracy: 19.27% 15 Validation loss: 2.655497 Best loss: 1.856821 Accuracy: 20.91% 16 Validation loss: 1.813184 Best loss: 1.813184 Accuracy: 22.01% 17 Validation loss: 2.527437 Best loss: 1.813184 Accuracy: 19.08% 18 Validation loss: 2.174712 Best loss: 1.813184 Accuracy: 19.08% 19 Validation loss: 2.524997 Best loss: 1.813184 Accuracy: 22.01% 20 Validation loss: 1.732186 Best loss: 1.732186 Accuracy: 18.73% 21 Validation loss: 2.199379 Best loss: 1.732186 Accuracy: 19.27% 22 Validation loss: 1.684675 Best loss: 1.684675 Accuracy: 18.73% 23 Validation loss: 2.217790 Best loss: 1.684675 Accuracy: 19.27% 24 Validation loss: 1.724423 Best loss: 1.684675 Accuracy: 18.73% 25 Validation loss: 1.744598 Best loss: 1.684675 Accuracy: 18.73% 26 Validation loss: 4.046068 Best loss: 1.684675 Accuracy: 19.08% 27 Validation loss: 2.406354 Best loss: 1.684675 Accuracy: 20.91% 28 Validation loss: 2.151017 Best loss: 1.684675 Accuracy: 19.27% 29 Validation loss: 1.936028 Best loss: 1.684675 Accuracy: 19.27% 30 Validation loss: 2.713347 Best loss: 1.684675 Accuracy: 22.01% 31 Validation loss: 2.196448 Best loss: 1.684675 Accuracy: 19.08% 32 Validation loss: 2.163520 Best loss: 1.684675 Accuracy: 18.73% 33 Validation loss: 3.707341 Best loss: 1.684675 Accuracy: 19.08% 34 Validation loss: 2.297056 Best loss: 1.684675 Accuracy: 20.91% 35 Validation loss: 3.078794 Best loss: 1.684675 Accuracy: 20.91% 36 Validation loss: 3.371657 Best loss: 1.684675 Accuracy: 19.27% 37 Validation loss: 3.341263 Best loss: 1.684675 Accuracy: 20.91% 38 Validation loss: 1.928011 Best loss: 1.684675 Accuracy: 19.08% 39 Validation loss: 4.551029 Best loss: 1.684675 Accuracy: 18.73% 40 Validation loss: 2.691634 Best loss: 1.684675 Accuracy: 18.73% 41 Validation loss: 4.253291 Best loss: 1.684675 Accuracy: 19.08% 42 Validation loss: 3.018767 Best loss: 1.684675 Accuracy: 20.91% 43 Validation loss: 3.032555 Best loss: 1.684675 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3, total= 3.7min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.722185 Best loss: 1.722185 Accuracy: 22.01% 1 Validation loss: 2.322997 Best loss: 1.722185 Accuracy: 22.01% 2 Validation loss: 2.228067 Best loss: 1.722185 Accuracy: 19.08% 3 Validation loss: 2.154531 Best loss: 1.722185 Accuracy: 22.01% 4 Validation loss: 3.231035 Best loss: 1.722185 Accuracy: 20.91% 5 Validation loss: 3.172793 Best loss: 1.722185 Accuracy: 20.91% 6 Validation loss: 2.639421 Best loss: 1.722185 Accuracy: 19.27% 7 Validation loss: 2.037729 Best loss: 1.722185 Accuracy: 22.01% 8 Validation loss: 1.745531 Best loss: 1.722185 Accuracy: 22.01% 9 Validation loss: 2.754615 Best loss: 1.722185 Accuracy: 18.73% 10 Validation loss: 1.974547 Best loss: 1.722185 Accuracy: 19.08% 11 Validation loss: 2.213611 Best loss: 1.722185 Accuracy: 18.73% 12 Validation loss: 2.587590 Best loss: 1.722185 Accuracy: 18.73% 13 Validation loss: 3.949859 Best loss: 1.722185 Accuracy: 19.08% 14 Validation loss: 1.798083 Best loss: 1.722185 Accuracy: 22.01% 15 Validation loss: 1.755204 Best loss: 1.722185 Accuracy: 22.01% 16 Validation loss: 2.001456 Best loss: 1.722185 Accuracy: 20.91% 17 Validation loss: 2.277148 Best loss: 1.722185 Accuracy: 18.73% 18 Validation loss: 1.954164 Best loss: 1.722185 Accuracy: 19.27% 19 Validation loss: 3.125824 Best loss: 1.722185 Accuracy: 22.01% 20 Validation loss: 1.882366 Best loss: 1.722185 Accuracy: 20.91% 21 Validation loss: 2.037230 Best loss: 1.722185 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=10, learning_rate=0.05, dropout_rate=0.3, total= 1.9min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.610959 Best loss: 1.610959 Accuracy: 19.08% 1 Validation loss: 1.613127 Best loss: 1.610959 Accuracy: 19.27% 2 Validation loss: 1.609386 Best loss: 1.609386 Accuracy: 22.01% 3 Validation loss: 1.609801 Best loss: 1.609386 Accuracy: 22.01% 4 Validation loss: 1.618811 Best loss: 1.609386 Accuracy: 22.01% 5 Validation loss: 1.612577 Best loss: 1.609386 Accuracy: 19.27% 6 Validation loss: 1.609906 Best loss: 1.609386 Accuracy: 22.01% 7 Validation loss: 1.615119 Best loss: 1.609386 Accuracy: 22.01% 8 Validation loss: 1.621897 Best loss: 1.609386 Accuracy: 22.01% 9 Validation loss: 1.610790 Best loss: 1.609386 Accuracy: 22.01% 10 Validation loss: 1.608212 Best loss: 1.608212 Accuracy: 20.91% 11 Validation loss: 1.619425 Best loss: 1.608212 Accuracy: 19.27% 12 Validation loss: 1.608469 Best loss: 1.608212 Accuracy: 20.91% 13 Validation loss: 1.618584 Best loss: 1.608212 Accuracy: 19.08% 14 Validation loss: 1.613560 Best loss: 1.608212 Accuracy: 18.73% 15 Validation loss: 1.611252 Best loss: 1.608212 Accuracy: 22.01% 16 Validation loss: 1.611343 Best loss: 1.608212 Accuracy: 19.08% 17 Validation loss: 1.610993 Best loss: 1.608212 Accuracy: 19.27% 18 Validation loss: 1.614764 Best loss: 1.608212 Accuracy: 18.73% 19 Validation loss: 1.610590 Best loss: 1.608212 Accuracy: 19.27% 20 Validation loss: 1.611319 Best loss: 1.608212 Accuracy: 20.91% 21 Validation loss: 1.614842 Best loss: 1.608212 Accuracy: 19.27% 22 Validation loss: 1.616512 Best loss: 1.608212 Accuracy: 19.27% 23 Validation loss: 1.618055 Best loss: 1.608212 Accuracy: 19.27% 24 Validation loss: 1.614491 Best loss: 1.608212 Accuracy: 22.01% 25 Validation loss: 1.609586 Best loss: 1.608212 Accuracy: 22.01% 26 Validation loss: 1.610508 Best loss: 1.608212 Accuracy: 22.01% 27 Validation loss: 1.612528 Best loss: 1.608212 Accuracy: 19.08% 28 Validation loss: 1.610706 Best loss: 1.608212 Accuracy: 19.08% 29 Validation loss: 1.614641 Best loss: 1.608212 Accuracy: 18.73% 30 Validation loss: 1.622547 Best loss: 1.608212 Accuracy: 19.27% 31 Validation loss: 1.608783 Best loss: 1.608212 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 18.8s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.609783 Best loss: 1.609783 Accuracy: 22.01% 1 Validation loss: 1.610492 Best loss: 1.609783 Accuracy: 19.08% 2 Validation loss: 1.615127 Best loss: 1.609783 Accuracy: 19.27% 3 Validation loss: 1.624164 Best loss: 1.609783 Accuracy: 18.73% 4 Validation loss: 1.614256 Best loss: 1.609783 Accuracy: 22.01% 5 Validation loss: 1.613982 Best loss: 1.609783 Accuracy: 19.27% 6 Validation loss: 1.614733 Best loss: 1.609783 Accuracy: 19.27% 7 Validation loss: 1.611901 Best loss: 1.609783 Accuracy: 22.01% 8 Validation loss: 1.609718 Best loss: 1.609718 Accuracy: 20.91% 9 Validation loss: 1.609544 Best loss: 1.609544 Accuracy: 20.91% 10 Validation loss: 1.619043 Best loss: 1.609544 Accuracy: 22.01% 11 Validation loss: 1.612234 Best loss: 1.609544 Accuracy: 22.01% 12 Validation loss: 1.609188 Best loss: 1.609188 Accuracy: 22.01% 13 Validation loss: 1.613472 Best loss: 1.609188 Accuracy: 19.27% 14 Validation loss: 1.613039 Best loss: 1.609188 Accuracy: 19.27% 15 Validation loss: 1.612358 Best loss: 1.609188 Accuracy: 22.01% 16 Validation loss: 1.612103 Best loss: 1.609188 Accuracy: 22.01% 17 Validation loss: 1.618465 Best loss: 1.609188 Accuracy: 19.08% 18 Validation loss: 1.610903 Best loss: 1.609188 Accuracy: 22.01% 19 Validation loss: 1.610624 Best loss: 1.609188 Accuracy: 22.01% 20 Validation loss: 1.612264 Best loss: 1.609188 Accuracy: 19.08% 21 Validation loss: 1.610903 Best loss: 1.609188 Accuracy: 22.01% 22 Validation loss: 1.617964 Best loss: 1.609188 Accuracy: 22.01% 23 Validation loss: 1.616951 Best loss: 1.609188 Accuracy: 22.01% 24 Validation loss: 1.612185 Best loss: 1.609188 Accuracy: 22.01% 25 Validation loss: 1.615036 Best loss: 1.609188 Accuracy: 22.01% 26 Validation loss: 1.607682 Best loss: 1.607682 Accuracy: 22.01% 27 Validation loss: 1.611644 Best loss: 1.607682 Accuracy: 22.01% 28 Validation loss: 1.626018 Best loss: 1.607682 Accuracy: 18.73% 29 Validation loss: 1.609979 Best loss: 1.607682 Accuracy: 18.73% 30 Validation loss: 1.611090 Best loss: 1.607682 Accuracy: 19.08% 31 Validation loss: 1.611652 Best loss: 1.607682 Accuracy: 19.27% 32 Validation loss: 1.612803 Best loss: 1.607682 Accuracy: 18.73% 33 Validation loss: 1.608830 Best loss: 1.607682 Accuracy: 20.91% 34 Validation loss: 1.613910 Best loss: 1.607682 Accuracy: 22.01% 35 Validation loss: 1.610115 Best loss: 1.607682 Accuracy: 22.01% 36 Validation loss: 1.613157 Best loss: 1.607682 Accuracy: 20.91% 37 Validation loss: 1.609833 Best loss: 1.607682 Accuracy: 22.01% 38 Validation loss: 1.620759 Best loss: 1.607682 Accuracy: 22.01% 39 Validation loss: 1.623760 Best loss: 1.607682 Accuracy: 19.27% 40 Validation loss: 1.609532 Best loss: 1.607682 Accuracy: 22.01% 41 Validation loss: 1.609141 Best loss: 1.607682 Accuracy: 22.01% 42 Validation loss: 1.622814 Best loss: 1.607682 Accuracy: 19.27% 43 Validation loss: 1.617255 Best loss: 1.607682 Accuracy: 22.01% 44 Validation loss: 1.616077 Best loss: 1.607682 Accuracy: 19.08% 45 Validation loss: 1.611638 Best loss: 1.607682 Accuracy: 19.08% 46 Validation loss: 1.623886 Best loss: 1.607682 Accuracy: 18.73% 47 Validation loss: 1.611055 Best loss: 1.607682 Accuracy: 19.08% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 28.0s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.613968 Best loss: 1.613968 Accuracy: 22.01% 1 Validation loss: 1.609759 Best loss: 1.609759 Accuracy: 19.08% 2 Validation loss: 1.617328 Best loss: 1.609759 Accuracy: 22.01% 3 Validation loss: 1.620003 Best loss: 1.609759 Accuracy: 19.27% 4 Validation loss: 1.609395 Best loss: 1.609395 Accuracy: 22.01% 5 Validation loss: 1.609368 Best loss: 1.609368 Accuracy: 22.01% 6 Validation loss: 1.615014 Best loss: 1.609368 Accuracy: 22.01% 7 Validation loss: 1.616839 Best loss: 1.609368 Accuracy: 18.73% 8 Validation loss: 1.611633 Best loss: 1.609368 Accuracy: 19.27% 9 Validation loss: 1.614371 Best loss: 1.609368 Accuracy: 18.73% 10 Validation loss: 1.612539 Best loss: 1.609368 Accuracy: 18.73% 11 Validation loss: 1.623439 Best loss: 1.609368 Accuracy: 22.01% 12 Validation loss: 1.608297 Best loss: 1.608297 Accuracy: 22.01% 13 Validation loss: 1.611835 Best loss: 1.608297 Accuracy: 19.27% 14 Validation loss: 1.608474 Best loss: 1.608297 Accuracy: 20.91% 15 Validation loss: 1.617344 Best loss: 1.608297 Accuracy: 19.08% 16 Validation loss: 1.618887 Best loss: 1.608297 Accuracy: 18.73% 17 Validation loss: 1.615229 Best loss: 1.608297 Accuracy: 19.08% 18 Validation loss: 1.608074 Best loss: 1.608074 Accuracy: 20.91% 19 Validation loss: 1.608634 Best loss: 1.608074 Accuracy: 22.01% 20 Validation loss: 1.620146 Best loss: 1.608074 Accuracy: 19.27% 21 Validation loss: 1.611821 Best loss: 1.608074 Accuracy: 22.01% 22 Validation loss: 1.613172 Best loss: 1.608074 Accuracy: 22.01% 23 Validation loss: 1.613918 Best loss: 1.608074 Accuracy: 22.01% 24 Validation loss: 1.613870 Best loss: 1.608074 Accuracy: 20.91% 25 Validation loss: 1.617802 Best loss: 1.608074 Accuracy: 22.01% 26 Validation loss: 1.613063 Best loss: 1.608074 Accuracy: 22.01% 27 Validation loss: 1.614678 Best loss: 1.608074 Accuracy: 19.27% 28 Validation loss: 1.630545 Best loss: 1.608074 Accuracy: 18.73% 29 Validation loss: 1.611471 Best loss: 1.608074 Accuracy: 22.01% 30 Validation loss: 1.611858 Best loss: 1.608074 Accuracy: 19.08% 31 Validation loss: 1.611840 Best loss: 1.608074 Accuracy: 22.01% 32 Validation loss: 1.615410 Best loss: 1.608074 Accuracy: 18.73% 33 Validation loss: 1.608509 Best loss: 1.608074 Accuracy: 22.01% 34 Validation loss: 1.613687 Best loss: 1.608074 Accuracy: 22.01% 35 Validation loss: 1.609653 Best loss: 1.608074 Accuracy: 19.27% 36 Validation loss: 1.609913 Best loss: 1.608074 Accuracy: 18.73% 37 Validation loss: 1.608673 Best loss: 1.608074 Accuracy: 22.01% 38 Validation loss: 1.617390 Best loss: 1.608074 Accuracy: 19.27% 39 Validation loss: 1.625790 Best loss: 1.608074 Accuracy: 19.27% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=70, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 23.7s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.319801 Best loss: 1.319801 Accuracy: 37.45% 1 Validation loss: 1.274088 Best loss: 1.274088 Accuracy: 41.40% 2 Validation loss: 1.245165 Best loss: 1.245165 Accuracy: 41.13% 3 Validation loss: 1.275528 Best loss: 1.245165 Accuracy: 40.62% 4 Validation loss: 1.375784 Best loss: 1.245165 Accuracy: 42.18% 5 Validation loss: 1.324490 Best loss: 1.245165 Accuracy: 42.14% 6 Validation loss: 1.248606 Best loss: 1.245165 Accuracy: 41.91% 7 Validation loss: 1.268378 Best loss: 1.245165 Accuracy: 41.91% 8 Validation loss: 1.257732 Best loss: 1.245165 Accuracy: 41.91% 9 Validation loss: 1.240199 Best loss: 1.240199 Accuracy: 41.91% 10 Validation loss: 1.249951 Best loss: 1.240199 Accuracy: 41.91% 11 Validation loss: 1.245262 Best loss: 1.240199 Accuracy: 41.91% 12 Validation loss: 1.257190 Best loss: 1.240199 Accuracy: 41.91% 13 Validation loss: 1.261016 Best loss: 1.240199 Accuracy: 38.58% 14 Validation loss: 1.248380 Best loss: 1.240199 Accuracy: 38.51% 15 Validation loss: 1.554072 Best loss: 1.240199 Accuracy: 22.01% 16 Validation loss: 1.431173 Best loss: 1.240199 Accuracy: 39.84% 17 Validation loss: 1.433104 Best loss: 1.240199 Accuracy: 37.26% 18 Validation loss: 1.446941 Best loss: 1.240199 Accuracy: 34.99% 19 Validation loss: 1.460270 Best loss: 1.240199 Accuracy: 39.84% 20 Validation loss: 1.634587 Best loss: 1.240199 Accuracy: 22.01% 21 Validation loss: 1.672305 Best loss: 1.240199 Accuracy: 19.27% 22 Validation loss: 1.618058 Best loss: 1.240199 Accuracy: 19.27% 23 Validation loss: 1.620102 Best loss: 1.240199 Accuracy: 19.27% 24 Validation loss: 1.619826 Best loss: 1.240199 Accuracy: 22.01% 25 Validation loss: 1.611761 Best loss: 1.240199 Accuracy: 22.01% 26 Validation loss: 1.618483 Best loss: 1.240199 Accuracy: 22.01% 27 Validation loss: 1.614303 Best loss: 1.240199 Accuracy: 19.08% 28 Validation loss: 1.609038 Best loss: 1.240199 Accuracy: 20.91% 29 Validation loss: 1.611783 Best loss: 1.240199 Accuracy: 18.73% 30 Validation loss: 1.617963 Best loss: 1.240199 Accuracy: 19.27% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3, total= 34.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.406138 Best loss: 1.406138 Accuracy: 31.04% 1 Validation loss: 1.463372 Best loss: 1.406138 Accuracy: 27.99% 2 Validation loss: 1.611727 Best loss: 1.406138 Accuracy: 19.27% 3 Validation loss: 1.619802 Best loss: 1.406138 Accuracy: 18.73% 4 Validation loss: 1.612870 Best loss: 1.406138 Accuracy: 22.01% 5 Validation loss: 1.610886 Best loss: 1.406138 Accuracy: 22.01% 6 Validation loss: 1.616850 Best loss: 1.406138 Accuracy: 19.27% 7 Validation loss: 1.613213 Best loss: 1.406138 Accuracy: 22.01% 8 Validation loss: 1.609157 Best loss: 1.406138 Accuracy: 22.01% 9 Validation loss: 1.608320 Best loss: 1.406138 Accuracy: 22.01% 10 Validation loss: 1.614522 Best loss: 1.406138 Accuracy: 22.01% 11 Validation loss: 1.610574 Best loss: 1.406138 Accuracy: 22.01% 12 Validation loss: 1.609135 Best loss: 1.406138 Accuracy: 22.01% 13 Validation loss: 1.612341 Best loss: 1.406138 Accuracy: 19.27% 14 Validation loss: 1.609190 Best loss: 1.406138 Accuracy: 22.01% 15 Validation loss: 1.610180 Best loss: 1.406138 Accuracy: 22.01% 16 Validation loss: 1.613044 Best loss: 1.406138 Accuracy: 22.01% 17 Validation loss: 1.615458 Best loss: 1.406138 Accuracy: 19.08% 18 Validation loss: 1.610242 Best loss: 1.406138 Accuracy: 22.01% 19 Validation loss: 1.609070 Best loss: 1.406138 Accuracy: 22.01% 20 Validation loss: 1.611214 Best loss: 1.406138 Accuracy: 22.01% 21 Validation loss: 1.612875 Best loss: 1.406138 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3, total= 24.5s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.609033 Best loss: 1.609033 Accuracy: 22.05% 1 Validation loss: 1.608654 Best loss: 1.608654 Accuracy: 22.13% 2 Validation loss: 1.615705 Best loss: 1.608654 Accuracy: 22.01% 3 Validation loss: 1.622234 Best loss: 1.608654 Accuracy: 19.27% 4 Validation loss: 1.608044 Best loss: 1.608044 Accuracy: 22.01% 5 Validation loss: 1.610939 Best loss: 1.608044 Accuracy: 22.01% 6 Validation loss: 1.612941 Best loss: 1.608044 Accuracy: 22.01% 7 Validation loss: 1.615310 Best loss: 1.608044 Accuracy: 22.01% 8 Validation loss: 1.609192 Best loss: 1.608044 Accuracy: 22.01% 9 Validation loss: 1.611430 Best loss: 1.608044 Accuracy: 22.01% 10 Validation loss: 1.612139 Best loss: 1.608044 Accuracy: 18.73% 11 Validation loss: 1.617147 Best loss: 1.608044 Accuracy: 22.01% 12 Validation loss: 1.608020 Best loss: 1.608020 Accuracy: 22.01% 13 Validation loss: 1.614054 Best loss: 1.608020 Accuracy: 19.27% 14 Validation loss: 1.609036 Best loss: 1.608020 Accuracy: 20.91% 15 Validation loss: 1.613213 Best loss: 1.608020 Accuracy: 22.01% 16 Validation loss: 1.620483 Best loss: 1.608020 Accuracy: 18.73% 17 Validation loss: 1.614961 Best loss: 1.608020 Accuracy: 19.08% 18 Validation loss: 1.608522 Best loss: 1.608020 Accuracy: 20.91% 19 Validation loss: 1.609453 Best loss: 1.608020 Accuracy: 22.01% 20 Validation loss: 1.617041 Best loss: 1.608020 Accuracy: 19.27% 21 Validation loss: 1.612379 Best loss: 1.608020 Accuracy: 22.01% 22 Validation loss: 1.613456 Best loss: 1.608020 Accuracy: 22.01% 23 Validation loss: 1.611028 Best loss: 1.608020 Accuracy: 22.01% 24 Validation loss: 1.610688 Best loss: 1.608020 Accuracy: 20.91% 25 Validation loss: 1.614551 Best loss: 1.608020 Accuracy: 22.01% 26 Validation loss: 1.609984 Best loss: 1.608020 Accuracy: 22.01% 27 Validation loss: 1.611451 Best loss: 1.608020 Accuracy: 22.01% 28 Validation loss: 1.625296 Best loss: 1.608020 Accuracy: 18.73% 29 Validation loss: 1.611936 Best loss: 1.608020 Accuracy: 22.01% 30 Validation loss: 1.610434 Best loss: 1.608020 Accuracy: 19.08% 31 Validation loss: 1.609723 Best loss: 1.608020 Accuracy: 22.01% 32 Validation loss: 1.612844 Best loss: 1.608020 Accuracy: 18.73% 33 Validation loss: 1.609556 Best loss: 1.608020 Accuracy: 22.01% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=100, batch_size=50, learning_rate=0.05, dropout_rate=0.3, total= 38.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.376199 Best loss: 0.376199 Accuracy: 91.87% 1 Validation loss: 0.313967 Best loss: 0.313967 Accuracy: 92.06% 2 Validation loss: 0.350038 Best loss: 0.313967 Accuracy: 93.12% 3 Validation loss: 0.338154 Best loss: 0.313967 Accuracy: 91.99% 4 Validation loss: 0.269337 Best loss: 0.269337 Accuracy: 93.71% 5 Validation loss: 0.315204 Best loss: 0.269337 Accuracy: 92.85% 6 Validation loss: 0.333477 Best loss: 0.269337 Accuracy: 92.42% 7 Validation loss: 0.407480 Best loss: 0.269337 Accuracy: 91.24% 8 Validation loss: 0.489458 Best loss: 0.269337 Accuracy: 87.18% 9 Validation loss: 0.445822 Best loss: 0.269337 Accuracy: 88.70% 10 Validation loss: 0.416503 Best loss: 0.269337 Accuracy: 91.40% 11 Validation loss: 0.460309 Best loss: 0.269337 Accuracy: 90.34% 12 Validation loss: 0.576501 Best loss: 0.269337 Accuracy: 84.75% 13 Validation loss: 0.887702 Best loss: 0.269337 Accuracy: 64.74% 14 Validation loss: 0.770059 Best loss: 0.269337 Accuracy: 75.76% 15 Validation loss: 0.797093 Best loss: 0.269337 Accuracy: 72.09% 16 Validation loss: 0.758941 Best loss: 0.269337 Accuracy: 75.14% 17 Validation loss: 0.726608 Best loss: 0.269337 Accuracy: 78.11% 18 Validation loss: 0.878936 Best loss: 0.269337 Accuracy: 65.13% 19 Validation loss: 1.028606 Best loss: 0.269337 Accuracy: 56.45% 20 Validation loss: 1.010196 Best loss: 0.269337 Accuracy: 57.39% 21 Validation loss: 0.966382 Best loss: 0.269337 Accuracy: 55.43% 22 Validation loss: 1.207306 Best loss: 0.269337 Accuracy: 44.68% 23 Validation loss: 1.320294 Best loss: 0.269337 Accuracy: 34.83% 24 Validation loss: 1.308442 Best loss: 0.269337 Accuracy: 38.58% 25 Validation loss: 1.364085 Best loss: 0.269337 Accuracy: 36.71% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6, total= 29.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.343791 Best loss: 0.343791 Accuracy: 92.77% 1 Validation loss: 0.400366 Best loss: 0.343791 Accuracy: 92.26% 2 Validation loss: 0.419304 Best loss: 0.343791 Accuracy: 90.66% 3 Validation loss: 0.397480 Best loss: 0.343791 Accuracy: 89.41% 4 Validation loss: 0.448109 Best loss: 0.343791 Accuracy: 88.47% 5 Validation loss: 0.539357 Best loss: 0.343791 Accuracy: 82.53% 6 Validation loss: 0.466653 Best loss: 0.343791 Accuracy: 88.00% 7 Validation loss: 0.724975 Best loss: 0.343791 Accuracy: 73.53% 8 Validation loss: 0.558049 Best loss: 0.343791 Accuracy: 81.00% 9 Validation loss: 0.523917 Best loss: 0.343791 Accuracy: 82.88% 10 Validation loss: 0.515921 Best loss: 0.343791 Accuracy: 85.57% 11 Validation loss: 0.441059 Best loss: 0.343791 Accuracy: 89.44% 12 Validation loss: 0.477910 Best loss: 0.343791 Accuracy: 88.04% 13 Validation loss: 0.643118 Best loss: 0.343791 Accuracy: 78.50% 14 Validation loss: 0.674212 Best loss: 0.343791 Accuracy: 74.98% 15 Validation loss: 0.673958 Best loss: 0.343791 Accuracy: 73.46% 16 Validation loss: 0.721355 Best loss: 0.343791 Accuracy: 71.85% 17 Validation loss: 0.730760 Best loss: 0.343791 Accuracy: 71.54% 18 Validation loss: 0.682554 Best loss: 0.343791 Accuracy: 73.06% 19 Validation loss: 0.731608 Best loss: 0.343791 Accuracy: 71.46% 20 Validation loss: 0.636718 Best loss: 0.343791 Accuracy: 73.85% 21 Validation loss: 0.658613 Best loss: 0.343791 Accuracy: 74.12% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6, total= 25.4s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.453847 Best loss: 0.453847 Accuracy: 88.51% 1 Validation loss: 0.288377 Best loss: 0.288377 Accuracy: 93.12% 2 Validation loss: 0.360330 Best loss: 0.288377 Accuracy: 91.95% 3 Validation loss: 0.491382 Best loss: 0.288377 Accuracy: 91.44% 4 Validation loss: 0.501503 Best loss: 0.288377 Accuracy: 90.11% 5 Validation loss: 0.753890 Best loss: 0.288377 Accuracy: 73.96% 6 Validation loss: 0.687996 Best loss: 0.288377 Accuracy: 72.71% 7 Validation loss: 0.660754 Best loss: 0.288377 Accuracy: 73.92% 8 Validation loss: 0.889139 Best loss: 0.288377 Accuracy: 58.25% 9 Validation loss: 0.766261 Best loss: 0.288377 Accuracy: 69.59% 10 Validation loss: 0.900737 Best loss: 0.288377 Accuracy: 57.11% 11 Validation loss: 0.970113 Best loss: 0.288377 Accuracy: 56.29% 12 Validation loss: 0.897538 Best loss: 0.288377 Accuracy: 57.43% 13 Validation loss: 1.047489 Best loss: 0.288377 Accuracy: 53.95% 14 Validation loss: 0.921795 Best loss: 0.288377 Accuracy: 56.72% 15 Validation loss: 0.959178 Best loss: 0.288377 Accuracy: 54.81% 16 Validation loss: 0.981416 Best loss: 0.288377 Accuracy: 54.65% 17 Validation loss: 1.003913 Best loss: 0.288377 Accuracy: 54.22% 18 Validation loss: 1.086074 Best loss: 0.288377 Accuracy: 49.53% 19 Validation loss: 1.078661 Best loss: 0.288377 Accuracy: 50.08% 20 Validation loss: 1.003265 Best loss: 0.288377 Accuracy: 54.10% 21 Validation loss: 1.050255 Best loss: 0.288377 Accuracy: 51.29% 22 Validation loss: 1.200408 Best loss: 0.288377 Accuracy: 41.56% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=50, learning_rate=0.01, dropout_rate=0.6, total= 25.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.622913 Best loss: 1.622913 Accuracy: 22.01% 1 Validation loss: 1.713925 Best loss: 1.622913 Accuracy: 19.27% 2 Validation loss: 1.642437 Best loss: 1.622913 Accuracy: 19.27% 3 Validation loss: 1.761912 Best loss: 1.622913 Accuracy: 19.27% 4 Validation loss: 1.714104 Best loss: 1.622913 Accuracy: 22.01% 5 Validation loss: 1.813097 Best loss: 1.622913 Accuracy: 18.73% 6 Validation loss: 1.677973 Best loss: 1.622913 Accuracy: 18.73% 7 Validation loss: 1.666575 Best loss: 1.622913 Accuracy: 20.91% 8 Validation loss: 1.732240 Best loss: 1.622913 Accuracy: 19.27% 9 Validation loss: 1.652136 Best loss: 1.622913 Accuracy: 18.73% 10 Validation loss: 1.673371 Best loss: 1.622913 Accuracy: 22.01% 11 Validation loss: 1.648985 Best loss: 1.622913 Accuracy: 22.01% 12 Validation loss: 1.646320 Best loss: 1.622913 Accuracy: 20.91% 13 Validation loss: 1.630677 Best loss: 1.622913 Accuracy: 19.27% 14 Validation loss: 1.754413 Best loss: 1.622913 Accuracy: 22.01% 15 Validation loss: 1.661756 Best loss: 1.622913 Accuracy: 22.01% 16 Validation loss: 1.638489 Best loss: 1.622913 Accuracy: 18.73% 17 Validation loss: 1.662441 Best loss: 1.622913 Accuracy: 18.73% 18 Validation loss: 1.704219 Best loss: 1.622913 Accuracy: 20.91% 19 Validation loss: 1.736790 Best loss: 1.622913 Accuracy: 22.01% 20 Validation loss: 1.633201 Best loss: 1.622913 Accuracy: 18.73% 21 Validation loss: 1.629899 Best loss: 1.622913 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 1.8min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.663486 Best loss: 1.663486 Accuracy: 22.01% 1 Validation loss: 1.618734 Best loss: 1.618734 Accuracy: 18.73% 2 Validation loss: 1.669722 Best loss: 1.618734 Accuracy: 19.27% 3 Validation loss: 1.683621 Best loss: 1.618734 Accuracy: 22.01% 4 Validation loss: 1.685020 Best loss: 1.618734 Accuracy: 22.01% 5 Validation loss: 1.658030 Best loss: 1.618734 Accuracy: 20.91% 6 Validation loss: 1.647170 Best loss: 1.618734 Accuracy: 19.27% 7 Validation loss: 1.675458 Best loss: 1.618734 Accuracy: 19.27% 8 Validation loss: 1.639526 Best loss: 1.618734 Accuracy: 22.01% 9 Validation loss: 1.610724 Best loss: 1.610724 Accuracy: 22.01% 10 Validation loss: 1.631784 Best loss: 1.610724 Accuracy: 22.01% 11 Validation loss: 1.644648 Best loss: 1.610724 Accuracy: 20.91% 12 Validation loss: 1.654162 Best loss: 1.610724 Accuracy: 20.91% 13 Validation loss: 1.640290 Best loss: 1.610724 Accuracy: 22.01% 14 Validation loss: 1.884427 Best loss: 1.610724 Accuracy: 18.73% 15 Validation loss: 1.725730 Best loss: 1.610724 Accuracy: 18.73% 16 Validation loss: 1.652587 Best loss: 1.610724 Accuracy: 19.27% 17 Validation loss: 1.705925 Best loss: 1.610724 Accuracy: 19.08% 18 Validation loss: 1.724167 Best loss: 1.610724 Accuracy: 20.91% 19 Validation loss: 1.615430 Best loss: 1.610724 Accuracy: 20.91% 20 Validation loss: 1.626318 Best loss: 1.610724 Accuracy: 20.91% 21 Validation loss: 1.644180 Best loss: 1.610724 Accuracy: 22.01% 22 Validation loss: 1.621657 Best loss: 1.610724 Accuracy: 19.27% 23 Validation loss: 1.718072 Best loss: 1.610724 Accuracy: 22.01% 24 Validation loss: 1.683651 Best loss: 1.610724 Accuracy: 18.73% 25 Validation loss: 6.837007 Best loss: 1.610724 Accuracy: 22.01% 26 Validation loss: 1.676415 Best loss: 1.610724 Accuracy: 19.08% 27 Validation loss: 1.614865 Best loss: 1.610724 Accuracy: 22.01% 28 Validation loss: 1.671858 Best loss: 1.610724 Accuracy: 19.08% 29 Validation loss: 1.701350 Best loss: 1.610724 Accuracy: 22.01% 30 Validation loss: 1.620001 Best loss: 1.610724 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 2.7min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.614338 Best loss: 1.614338 Accuracy: 18.73% 1 Validation loss: 1.691362 Best loss: 1.614338 Accuracy: 19.27% 2 Validation loss: 1.620174 Best loss: 1.614338 Accuracy: 19.27% 3 Validation loss: 1.672203 Best loss: 1.614338 Accuracy: 19.08% 4 Validation loss: 1.753689 Best loss: 1.614338 Accuracy: 19.08% 5 Validation loss: 1.664109 Best loss: 1.614338 Accuracy: 20.91% 6 Validation loss: 1.665451 Best loss: 1.614338 Accuracy: 19.27% 7 Validation loss: 1.647918 Best loss: 1.614338 Accuracy: 18.73% 8 Validation loss: 1.683239 Best loss: 1.614338 Accuracy: 18.73% 9 Validation loss: 1.714252 Best loss: 1.614338 Accuracy: 22.01% 10 Validation loss: 1.627946 Best loss: 1.614338 Accuracy: 18.73% 11 Validation loss: 1.704932 Best loss: 1.614338 Accuracy: 18.73% 12 Validation loss: 1.713096 Best loss: 1.614338 Accuracy: 20.91% 13 Validation loss: 1.678811 Best loss: 1.614338 Accuracy: 22.01% 14 Validation loss: 1.854946 Best loss: 1.614338 Accuracy: 18.73% 15 Validation loss: 1.639386 Best loss: 1.614338 Accuracy: 22.01% 16 Validation loss: 1.645660 Best loss: 1.614338 Accuracy: 19.27% 17 Validation loss: 1.628370 Best loss: 1.614338 Accuracy: 20.91% 18 Validation loss: 1.698103 Best loss: 1.614338 Accuracy: 22.01% 19 Validation loss: 1.652491 Best loss: 1.614338 Accuracy: 19.27% 20 Validation loss: 1.640774 Best loss: 1.614338 Accuracy: 20.91% 21 Validation loss: 1.695997 Best loss: 1.614338 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 1.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 1.251853 Best loss: 1.251853 Accuracy: 53.36% 1 Validation loss: 1.632751 Best loss: 1.251853 Accuracy: 19.27% 2 Validation loss: 1.690061 Best loss: 1.251853 Accuracy: 19.27% 3 Validation loss: 1.712494 Best loss: 1.251853 Accuracy: 19.27% 4 Validation loss: 1.741684 Best loss: 1.251853 Accuracy: 18.73% 5 Validation loss: 1.691165 Best loss: 1.251853 Accuracy: 22.01% 6 Validation loss: 1.635431 Best loss: 1.251853 Accuracy: 19.08% 7 Validation loss: 1.761021 Best loss: 1.251853 Accuracy: 19.08% 8 Validation loss: 1.708401 Best loss: 1.251853 Accuracy: 19.27% 9 Validation loss: 1.843622 Best loss: 1.251853 Accuracy: 19.27% 10 Validation loss: 1.659562 Best loss: 1.251853 Accuracy: 18.73% 11 Validation loss: 1.739221 Best loss: 1.251853 Accuracy: 19.08% 12 Validation loss: 1.734355 Best loss: 1.251853 Accuracy: 19.08% 13 Validation loss: 1.650780 Best loss: 1.251853 Accuracy: 20.91% 14 Validation loss: 1.645068 Best loss: 1.251853 Accuracy: 18.73% 15 Validation loss: 1.669529 Best loss: 1.251853 Accuracy: 19.08% 16 Validation loss: 1.739544 Best loss: 1.251853 Accuracy: 18.73% 17 Validation loss: 1.649988 Best loss: 1.251853 Accuracy: 22.01% 18 Validation loss: 1.686872 Best loss: 1.251853 Accuracy: 20.91% 19 Validation loss: 1.631175 Best loss: 1.251853 Accuracy: 20.91% 20 Validation loss: 1.658821 Best loss: 1.251853 Accuracy: 19.27% 21 Validation loss: 1.759572 Best loss: 1.251853 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 24.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 1.613135 Best loss: 1.613135 Accuracy: 36.47% 1 Validation loss: 1.645113 Best loss: 1.613135 Accuracy: 19.08% 2 Validation loss: 1.662588 Best loss: 1.613135 Accuracy: 20.91% 3 Validation loss: 1.652616 Best loss: 1.613135 Accuracy: 22.01% 4 Validation loss: 1.666343 Best loss: 1.613135 Accuracy: 19.27% 5 Validation loss: 1.712007 Best loss: 1.613135 Accuracy: 20.91% 6 Validation loss: 1.654462 Best loss: 1.613135 Accuracy: 19.27% 7 Validation loss: 1.804651 Best loss: 1.613135 Accuracy: 18.73% 8 Validation loss: 1.826288 Best loss: 1.613135 Accuracy: 18.73% 9 Validation loss: 1.708499 Best loss: 1.613135 Accuracy: 18.73% 10 Validation loss: 1.672716 Best loss: 1.613135 Accuracy: 19.08% 11 Validation loss: 1.691086 Best loss: 1.613135 Accuracy: 22.01% 12 Validation loss: 1.859798 Best loss: 1.613135 Accuracy: 20.91% 13 Validation loss: 1.671651 Best loss: 1.613135 Accuracy: 19.08% 14 Validation loss: 1.639897 Best loss: 1.613135 Accuracy: 18.73% 15 Validation loss: 1.819149 Best loss: 1.613135 Accuracy: 19.08% 16 Validation loss: 1.666695 Best loss: 1.613135 Accuracy: 18.73% 17 Validation loss: 1.673237 Best loss: 1.613135 Accuracy: 19.27% 18 Validation loss: 1.648671 Best loss: 1.613135 Accuracy: 19.08% 19 Validation loss: 1.701077 Best loss: 1.613135 Accuracy: 19.27% 20 Validation loss: 1.700924 Best loss: 1.613135 Accuracy: 22.01% 21 Validation loss: 1.760052 Best loss: 1.613135 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 24.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.272536 Best loss: 0.272536 Accuracy: 93.94% 1 Validation loss: 1.644488 Best loss: 0.272536 Accuracy: 19.27% 2 Validation loss: 1.756182 Best loss: 0.272536 Accuracy: 20.91% 3 Validation loss: 1.700079 Best loss: 0.272536 Accuracy: 22.01% 4 Validation loss: 1.773024 Best loss: 0.272536 Accuracy: 19.27% 5 Validation loss: 1.708282 Best loss: 0.272536 Accuracy: 20.91% 6 Validation loss: 1.670462 Best loss: 0.272536 Accuracy: 19.27% 7 Validation loss: 1.774604 Best loss: 0.272536 Accuracy: 18.73% 8 Validation loss: 1.755814 Best loss: 0.272536 Accuracy: 22.01% 9 Validation loss: 1.902144 Best loss: 0.272536 Accuracy: 20.91% 10 Validation loss: 1.669155 Best loss: 0.272536 Accuracy: 18.73% 11 Validation loss: 1.892161 Best loss: 0.272536 Accuracy: 18.73% 12 Validation loss: 1.802391 Best loss: 0.272536 Accuracy: 20.91% 13 Validation loss: 1.619435 Best loss: 0.272536 Accuracy: 22.01% 14 Validation loss: 1.693270 Best loss: 0.272536 Accuracy: 19.08% 15 Validation loss: 1.656775 Best loss: 0.272536 Accuracy: 22.01% 16 Validation loss: 1.633002 Best loss: 0.272536 Accuracy: 22.01% 17 Validation loss: 1.664847 Best loss: 0.272536 Accuracy: 19.27% 18 Validation loss: 1.644787 Best loss: 0.272536 Accuracy: 19.08% 19 Validation loss: 1.743184 Best loss: 0.272536 Accuracy: 18.73% 20 Validation loss: 1.720337 Best loss: 0.272536 Accuracy: 18.73% 21 Validation loss: 1.819168 Best loss: 0.272536 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=50, learning_rate=0.02, dropout_rate=0.3, total= 24.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.673069 Best loss: 1.673069 Accuracy: 19.27% 1 Validation loss: 1.829582 Best loss: 1.673069 Accuracy: 19.08% 2 Validation loss: 1.754017 Best loss: 1.673069 Accuracy: 22.01% 3 Validation loss: 1.776856 Best loss: 1.673069 Accuracy: 18.73% 4 Validation loss: 1.758369 Best loss: 1.673069 Accuracy: 18.73% 5 Validation loss: 1.663307 Best loss: 1.663307 Accuracy: 20.91% 6 Validation loss: 1.703243 Best loss: 1.663307 Accuracy: 19.27% 7 Validation loss: 1.649201 Best loss: 1.649201 Accuracy: 22.01% 8 Validation loss: 2.027719 Best loss: 1.649201 Accuracy: 19.08% 9 Validation loss: 1.918205 Best loss: 1.649201 Accuracy: 19.27% 10 Validation loss: 2.655689 Best loss: 1.649201 Accuracy: 22.01% 11 Validation loss: 1.953923 Best loss: 1.649201 Accuracy: 19.08% 12 Validation loss: 1.875643 Best loss: 1.649201 Accuracy: 19.08% 13 Validation loss: 1.666909 Best loss: 1.649201 Accuracy: 19.08% 14 Validation loss: 2.082243 Best loss: 1.649201 Accuracy: 18.73% 15 Validation loss: 2.049284 Best loss: 1.649201 Accuracy: 18.73% 16 Validation loss: 2.007152 Best loss: 1.649201 Accuracy: 19.08% 17 Validation loss: 2.437870 Best loss: 1.649201 Accuracy: 18.73% 18 Validation loss: 2.126289 Best loss: 1.649201 Accuracy: 20.91% 19 Validation loss: 1.673189 Best loss: 1.649201 Accuracy: 22.01% 20 Validation loss: 1.895276 Best loss: 1.649201 Accuracy: 19.27% 21 Validation loss: 1.716326 Best loss: 1.649201 Accuracy: 22.01% 22 Validation loss: 1.969317 Best loss: 1.649201 Accuracy: 20.91% 23 Validation loss: 1.853645 Best loss: 1.649201 Accuracy: 20.91% 24 Validation loss: 1.701266 Best loss: 1.649201 Accuracy: 18.73% 25 Validation loss: 1.750096 Best loss: 1.649201 Accuracy: 22.01% 26 Validation loss: 1.614368 Best loss: 1.614368 Accuracy: 22.01% 27 Validation loss: 3.137303 Best loss: 1.614368 Accuracy: 19.27% 28 Validation loss: 1.944066 Best loss: 1.614368 Accuracy: 22.01% 29 Validation loss: 2.061138 Best loss: 1.614368 Accuracy: 18.73% 30 Validation loss: 2.037326 Best loss: 1.614368 Accuracy: 18.73% 31 Validation loss: 1.817108 Best loss: 1.614368 Accuracy: 20.91% 32 Validation loss: 1.671890 Best loss: 1.614368 Accuracy: 19.08% 33 Validation loss: 1.728474 Best loss: 1.614368 Accuracy: 20.91% 34 Validation loss: 1.907635 Best loss: 1.614368 Accuracy: 22.01% 35 Validation loss: 1.932810 Best loss: 1.614368 Accuracy: 20.91% 36 Validation loss: 2.393437 Best loss: 1.614368 Accuracy: 22.01% 37 Validation loss: 1.704121 Best loss: 1.614368 Accuracy: 18.73% 38 Validation loss: 2.535507 Best loss: 1.614368 Accuracy: 22.01% 39 Validation loss: 2.062447 Best loss: 1.614368 Accuracy: 19.27% 40 Validation loss: 1.733631 Best loss: 1.614368 Accuracy: 20.91% 41 Validation loss: 1.720416 Best loss: 1.614368 Accuracy: 18.73% 42 Validation loss: 1.683540 Best loss: 1.614368 Accuracy: 19.08% 43 Validation loss: 1.794480 Best loss: 1.614368 Accuracy: 20.91% 44 Validation loss: 2.170772 Best loss: 1.614368 Accuracy: 18.73% 45 Validation loss: 2.576666 Best loss: 1.614368 Accuracy: 19.08% 46 Validation loss: 1.961155 Best loss: 1.614368 Accuracy: 19.08% 47 Validation loss: 2.424041 Best loss: 1.614368 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 28.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.662189 Best loss: 1.662189 Accuracy: 22.01% 1 Validation loss: 1.653455 Best loss: 1.653455 Accuracy: 18.73% 2 Validation loss: 1.615953 Best loss: 1.615953 Accuracy: 22.01% 3 Validation loss: 1.637527 Best loss: 1.615953 Accuracy: 22.01% 4 Validation loss: 1.692917 Best loss: 1.615953 Accuracy: 19.08% 5 Validation loss: 1.647966 Best loss: 1.615953 Accuracy: 19.08% 6 Validation loss: 1.680476 Best loss: 1.615953 Accuracy: 18.73% 7 Validation loss: 1.730717 Best loss: 1.615953 Accuracy: 19.27% 8 Validation loss: 1.764365 Best loss: 1.615953 Accuracy: 22.01% 9 Validation loss: 1.797486 Best loss: 1.615953 Accuracy: 19.08% 10 Validation loss: 1.714033 Best loss: 1.615953 Accuracy: 22.01% 11 Validation loss: 1.685432 Best loss: 1.615953 Accuracy: 22.01% 12 Validation loss: 1.653650 Best loss: 1.615953 Accuracy: 19.27% 13 Validation loss: 1.935055 Best loss: 1.615953 Accuracy: 19.08% 14 Validation loss: 1.956234 Best loss: 1.615953 Accuracy: 19.08% 15 Validation loss: 1.685410 Best loss: 1.615953 Accuracy: 19.08% 16 Validation loss: 1.752678 Best loss: 1.615953 Accuracy: 22.01% 17 Validation loss: 1.695387 Best loss: 1.615953 Accuracy: 18.73% 18 Validation loss: 1.706126 Best loss: 1.615953 Accuracy: 19.27% 19 Validation loss: 1.656490 Best loss: 1.615953 Accuracy: 22.01% 20 Validation loss: 1.775720 Best loss: 1.615953 Accuracy: 18.73% 21 Validation loss: 1.786095 Best loss: 1.615953 Accuracy: 22.01% 22 Validation loss: 1.907646 Best loss: 1.615953 Accuracy: 18.73% 23 Validation loss: 1.645706 Best loss: 1.615953 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 14.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.865176 Best loss: 1.865176 Accuracy: 20.91% 1 Validation loss: 1.980201 Best loss: 1.865176 Accuracy: 19.08% 2 Validation loss: 1.664968 Best loss: 1.664968 Accuracy: 20.91% 3 Validation loss: 1.695557 Best loss: 1.664968 Accuracy: 22.01% 4 Validation loss: 2.246447 Best loss: 1.664968 Accuracy: 20.91% 5 Validation loss: 1.777914 Best loss: 1.664968 Accuracy: 20.91% 6 Validation loss: 1.943854 Best loss: 1.664968 Accuracy: 20.91% 7 Validation loss: 2.168597 Best loss: 1.664968 Accuracy: 18.73% 8 Validation loss: 1.966082 Best loss: 1.664968 Accuracy: 22.01% 9 Validation loss: 1.841778 Best loss: 1.664968 Accuracy: 19.08% 10 Validation loss: 1.840677 Best loss: 1.664968 Accuracy: 18.73% 11 Validation loss: 1.923024 Best loss: 1.664968 Accuracy: 19.27% 12 Validation loss: 2.242763 Best loss: 1.664968 Accuracy: 19.08% 13 Validation loss: 1.737965 Best loss: 1.664968 Accuracy: 22.01% 14 Validation loss: 2.031966 Best loss: 1.664968 Accuracy: 19.08% 15 Validation loss: 1.664840 Best loss: 1.664840 Accuracy: 22.01% 16 Validation loss: 2.516648 Best loss: 1.664840 Accuracy: 18.73% 17 Validation loss: 1.656894 Best loss: 1.656894 Accuracy: 22.01% 18 Validation loss: 1.752496 Best loss: 1.656894 Accuracy: 22.01% 19 Validation loss: 2.048165 Best loss: 1.656894 Accuracy: 18.73% 20 Validation loss: 1.650886 Best loss: 1.650886 Accuracy: 22.01% 21 Validation loss: 1.881698 Best loss: 1.650886 Accuracy: 22.01% 22 Validation loss: 2.484859 Best loss: 1.650886 Accuracy: 20.91% 23 Validation loss: 1.771033 Best loss: 1.650886 Accuracy: 22.01% 24 Validation loss: 1.715100 Best loss: 1.650886 Accuracy: 20.91% 25 Validation loss: 1.911318 Best loss: 1.650886 Accuracy: 22.01% 26 Validation loss: 1.747377 Best loss: 1.650886 Accuracy: 18.73% 27 Validation loss: 1.769615 Best loss: 1.650886 Accuracy: 19.27% 28 Validation loss: 1.910045 Best loss: 1.650886 Accuracy: 18.73% 29 Validation loss: 2.052456 Best loss: 1.650886 Accuracy: 19.27% 30 Validation loss: 2.584853 Best loss: 1.650886 Accuracy: 19.27% 31 Validation loss: 2.356132 Best loss: 1.650886 Accuracy: 22.01% 32 Validation loss: 2.005762 Best loss: 1.650886 Accuracy: 19.08% 33 Validation loss: 1.854444 Best loss: 1.650886 Accuracy: 18.73% 34 Validation loss: 2.226954 Best loss: 1.650886 Accuracy: 18.73% 35 Validation loss: 1.667116 Best loss: 1.650886 Accuracy: 18.73% 36 Validation loss: 2.037481 Best loss: 1.650886 Accuracy: 18.73% 37 Validation loss: 2.190071 Best loss: 1.650886 Accuracy: 19.08% 38 Validation loss: 1.867853 Best loss: 1.650886 Accuracy: 20.91% 39 Validation loss: 1.879594 Best loss: 1.650886 Accuracy: 20.91% 40 Validation loss: 1.716182 Best loss: 1.650886 Accuracy: 20.91% 41 Validation loss: 1.767133 Best loss: 1.650886 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=120, batch_size=100, learning_rate=0.1, dropout_rate=0.5, total= 24.4s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.425855 Best loss: 0.425855 Accuracy: 77.91% 1 Validation loss: 0.216674 Best loss: 0.216674 Accuracy: 95.04% 2 Validation loss: 0.134203 Best loss: 0.134203 Accuracy: 96.60% 3 Validation loss: 0.128278 Best loss: 0.128278 Accuracy: 96.44% 4 Validation loss: 0.117107 Best loss: 0.117107 Accuracy: 96.87% 5 Validation loss: 0.115305 Best loss: 0.115305 Accuracy: 97.03% 6 Validation loss: 0.121341 Best loss: 0.115305 Accuracy: 96.83% 7 Validation loss: 0.118874 Best loss: 0.115305 Accuracy: 96.68% 8 Validation loss: 0.117782 Best loss: 0.115305 Accuracy: 96.76% 9 Validation loss: 0.128086 Best loss: 0.115305 Accuracy: 97.11% 10 Validation loss: 0.122032 Best loss: 0.115305 Accuracy: 96.91% 11 Validation loss: 0.115362 Best loss: 0.115305 Accuracy: 97.15% 12 Validation loss: 0.113034 Best loss: 0.113034 Accuracy: 97.11% 13 Validation loss: 0.119335 Best loss: 0.113034 Accuracy: 96.99% 14 Validation loss: 0.126085 Best loss: 0.113034 Accuracy: 96.91% 15 Validation loss: 0.110626 Best loss: 0.110626 Accuracy: 97.19% 16 Validation loss: 0.123154 Best loss: 0.110626 Accuracy: 96.87% 17 Validation loss: 0.108321 Best loss: 0.108321 Accuracy: 97.15% 18 Validation loss: 0.110672 Best loss: 0.108321 Accuracy: 97.19% 19 Validation loss: 0.104182 Best loss: 0.104182 Accuracy: 97.15% 20 Validation loss: 0.113430 Best loss: 0.104182 Accuracy: 97.11% 21 Validation loss: 0.117869 Best loss: 0.104182 Accuracy: 97.03% 22 Validation loss: 0.114579 Best loss: 0.104182 Accuracy: 97.15% 23 Validation loss: 0.113516 Best loss: 0.104182 Accuracy: 97.15% 24 Validation loss: 0.113507 Best loss: 0.104182 Accuracy: 97.15% 25 Validation loss: 0.104303 Best loss: 0.104182 Accuracy: 97.19% 26 Validation loss: 0.111116 Best loss: 0.104182 Accuracy: 97.26% 27 Validation loss: 0.106400 Best loss: 0.104182 Accuracy: 96.95% 28 Validation loss: 0.110987 Best loss: 0.104182 Accuracy: 97.30% 29 Validation loss: 0.104225 Best loss: 0.104182 Accuracy: 97.22% 30 Validation loss: 0.109614 Best loss: 0.104182 Accuracy: 97.30% 31 Validation loss: 0.114947 Best loss: 0.104182 Accuracy: 97.15% 32 Validation loss: 0.117465 Best loss: 0.104182 Accuracy: 97.03% 33 Validation loss: 0.113150 Best loss: 0.104182 Accuracy: 97.15% 34 Validation loss: 0.109591 Best loss: 0.104182 Accuracy: 97.26% 35 Validation loss: 0.119260 Best loss: 0.104182 Accuracy: 97.38% 36 Validation loss: 0.111507 Best loss: 0.104182 Accuracy: 97.19% 37 Validation loss: 0.114030 Best loss: 0.104182 Accuracy: 97.03% 38 Validation loss: 0.124196 Best loss: 0.104182 Accuracy: 97.03% 39 Validation loss: 0.114204 Best loss: 0.104182 Accuracy: 97.03% 40 Validation loss: 0.107036 Best loss: 0.104182 Accuracy: 97.19% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2, total= 7.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.158857 Best loss: 0.158857 Accuracy: 95.74% 1 Validation loss: 0.143180 Best loss: 0.143180 Accuracy: 96.40% 2 Validation loss: 0.122275 Best loss: 0.122275 Accuracy: 96.64% 3 Validation loss: 0.131371 Best loss: 0.122275 Accuracy: 96.56% 4 Validation loss: 0.123488 Best loss: 0.122275 Accuracy: 96.72% 5 Validation loss: 0.130908 Best loss: 0.122275 Accuracy: 96.44% 6 Validation loss: 0.112869 Best loss: 0.112869 Accuracy: 97.11% 7 Validation loss: 0.116373 Best loss: 0.112869 Accuracy: 96.95% 8 Validation loss: 0.105110 Best loss: 0.105110 Accuracy: 97.11% 9 Validation loss: 0.105295 Best loss: 0.105110 Accuracy: 97.34% 10 Validation loss: 0.119826 Best loss: 0.105110 Accuracy: 96.72% 11 Validation loss: 0.134138 Best loss: 0.105110 Accuracy: 96.72% 12 Validation loss: 0.117716 Best loss: 0.105110 Accuracy: 97.03% 13 Validation loss: 0.117901 Best loss: 0.105110 Accuracy: 96.87% 14 Validation loss: 0.105362 Best loss: 0.105110 Accuracy: 97.07% 15 Validation loss: 0.118098 Best loss: 0.105110 Accuracy: 96.76% 16 Validation loss: 0.111448 Best loss: 0.105110 Accuracy: 96.91% 17 Validation loss: 0.111489 Best loss: 0.105110 Accuracy: 97.22% 18 Validation loss: 0.119271 Best loss: 0.105110 Accuracy: 96.83% 19 Validation loss: 0.113192 Best loss: 0.105110 Accuracy: 97.30% 20 Validation loss: 0.123803 Best loss: 0.105110 Accuracy: 96.99% 21 Validation loss: 0.097415 Best loss: 0.097415 Accuracy: 97.30% 22 Validation loss: 0.110756 Best loss: 0.097415 Accuracy: 97.26% 23 Validation loss: 0.110191 Best loss: 0.097415 Accuracy: 97.30% 24 Validation loss: 0.108788 Best loss: 0.097415 Accuracy: 97.03% 25 Validation loss: 0.118800 Best loss: 0.097415 Accuracy: 96.79% 26 Validation loss: 0.104379 Best loss: 0.097415 Accuracy: 97.19% 27 Validation loss: 0.109628 Best loss: 0.097415 Accuracy: 97.19% 28 Validation loss: 0.106905 Best loss: 0.097415 Accuracy: 97.15% 29 Validation loss: 0.112001 Best loss: 0.097415 Accuracy: 97.30% 30 Validation loss: 0.114947 Best loss: 0.097415 Accuracy: 97.26% 31 Validation loss: 0.108578 Best loss: 0.097415 Accuracy: 97.15% 32 Validation loss: 0.108157 Best loss: 0.097415 Accuracy: 97.30% 33 Validation loss: 0.111717 Best loss: 0.097415 Accuracy: 97.11% 34 Validation loss: 0.101413 Best loss: 0.097415 Accuracy: 97.42% 35 Validation loss: 0.108614 Best loss: 0.097415 Accuracy: 97.34% 36 Validation loss: 0.105621 Best loss: 0.097415 Accuracy: 97.26% 37 Validation loss: 0.106502 Best loss: 0.097415 Accuracy: 96.99% 38 Validation loss: 0.103465 Best loss: 0.097415 Accuracy: 97.50% 39 Validation loss: 0.106202 Best loss: 0.097415 Accuracy: 97.22% 40 Validation loss: 0.105442 Best loss: 0.097415 Accuracy: 97.46% 41 Validation loss: 0.110296 Best loss: 0.097415 Accuracy: 97.15% 42 Validation loss: 0.098545 Best loss: 0.097415 Accuracy: 97.38% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2, total= 7.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2 0 Validation loss: 0.148773 Best loss: 0.148773 Accuracy: 95.82% 1 Validation loss: 0.130664 Best loss: 0.130664 Accuracy: 96.68% 2 Validation loss: 0.106294 Best loss: 0.106294 Accuracy: 97.19% 3 Validation loss: 0.123471 Best loss: 0.106294 Accuracy: 96.79% 4 Validation loss: 0.117257 Best loss: 0.106294 Accuracy: 96.79% 5 Validation loss: 0.109352 Best loss: 0.106294 Accuracy: 97.26% 6 Validation loss: 0.107334 Best loss: 0.106294 Accuracy: 96.99% 7 Validation loss: 0.109835 Best loss: 0.106294 Accuracy: 97.11% 8 Validation loss: 0.115049 Best loss: 0.106294 Accuracy: 97.26% 9 Validation loss: 0.100758 Best loss: 0.100758 Accuracy: 97.65% 10 Validation loss: 0.109147 Best loss: 0.100758 Accuracy: 97.11% 11 Validation loss: 0.121331 Best loss: 0.100758 Accuracy: 97.03% 12 Validation loss: 0.122329 Best loss: 0.100758 Accuracy: 96.99% 13 Validation loss: 0.101589 Best loss: 0.100758 Accuracy: 97.38% 14 Validation loss: 0.102301 Best loss: 0.100758 Accuracy: 97.50% 15 Validation loss: 0.107594 Best loss: 0.100758 Accuracy: 97.26% 16 Validation loss: 0.113853 Best loss: 0.100758 Accuracy: 97.38% 17 Validation loss: 0.103851 Best loss: 0.100758 Accuracy: 97.26% 18 Validation loss: 0.108726 Best loss: 0.100758 Accuracy: 97.26% 19 Validation loss: 0.107097 Best loss: 0.100758 Accuracy: 97.46% 20 Validation loss: 0.109845 Best loss: 0.100758 Accuracy: 97.15% 21 Validation loss: 0.099780 Best loss: 0.099780 Accuracy: 97.38% 22 Validation loss: 0.100231 Best loss: 0.099780 Accuracy: 97.46% 23 Validation loss: 0.101915 Best loss: 0.099780 Accuracy: 97.73% 24 Validation loss: 0.107178 Best loss: 0.099780 Accuracy: 97.34% 25 Validation loss: 0.097962 Best loss: 0.097962 Accuracy: 97.69% 26 Validation loss: 0.100320 Best loss: 0.097962 Accuracy: 97.42% 27 Validation loss: 0.104355 Best loss: 0.097962 Accuracy: 97.58% 28 Validation loss: 0.104419 Best loss: 0.097962 Accuracy: 97.50% 29 Validation loss: 0.111678 Best loss: 0.097962 Accuracy: 97.22% 30 Validation loss: 0.113286 Best loss: 0.097962 Accuracy: 97.22% 31 Validation loss: 0.102039 Best loss: 0.097962 Accuracy: 97.30% 32 Validation loss: 0.107669 Best loss: 0.097962 Accuracy: 97.58% 33 Validation loss: 0.099315 Best loss: 0.097962 Accuracy: 97.46% 34 Validation loss: 0.114832 Best loss: 0.097962 Accuracy: 97.22% 35 Validation loss: 0.104947 Best loss: 0.097962 Accuracy: 97.50% 36 Validation loss: 0.097641 Best loss: 0.097641 Accuracy: 97.50% 37 Validation loss: 0.097628 Best loss: 0.097628 Accuracy: 97.62% 38 Validation loss: 0.104203 Best loss: 0.097628 Accuracy: 97.73% 39 Validation loss: 0.104688 Best loss: 0.097628 Accuracy: 97.50% 40 Validation loss: 0.098547 Best loss: 0.097628 Accuracy: 97.62% 41 Validation loss: 0.100081 Best loss: 0.097628 Accuracy: 97.54% 42 Validation loss: 0.098928 Best loss: 0.097628 Accuracy: 97.65% 43 Validation loss: 0.107404 Best loss: 0.097628 Accuracy: 97.22% 44 Validation loss: 0.100576 Best loss: 0.097628 Accuracy: 97.93% 45 Validation loss: 0.103298 Best loss: 0.097628 Accuracy: 97.69% 46 Validation loss: 0.097777 Best loss: 0.097628 Accuracy: 97.81% 47 Validation loss: 0.106432 Best loss: 0.097628 Accuracy: 97.58% 48 Validation loss: 0.110727 Best loss: 0.097628 Accuracy: 97.34% 49 Validation loss: 0.102805 Best loss: 0.097628 Accuracy: 97.54% 50 Validation loss: 0.099578 Best loss: 0.097628 Accuracy: 97.85% 51 Validation loss: 0.107437 Best loss: 0.097628 Accuracy: 97.58% 52 Validation loss: 0.110570 Best loss: 0.097628 Accuracy: 97.38% 53 Validation loss: 0.112959 Best loss: 0.097628 Accuracy: 97.19% 54 Validation loss: 0.113321 Best loss: 0.097628 Accuracy: 97.46% 55 Validation loss: 0.102650 Best loss: 0.097628 Accuracy: 97.58% 56 Validation loss: 0.098147 Best loss: 0.097628 Accuracy: 97.58% 57 Validation loss: 0.100814 Best loss: 0.097628 Accuracy: 97.62% 58 Validation loss: 0.099286 Best loss: 0.097628 Accuracy: 97.77% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=10, batch_size=500, learning_rate=0.02, dropout_rate=0.2, total= 10.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 1412.347290 Best loss: 1412.347290 Accuracy: 93.71% 1 Validation loss: 400980.718750 Best loss: 1412.347290 Accuracy: 94.02% 2 Validation loss: 52833.507812 Best loss: 1412.347290 Accuracy: 95.82% 3 Validation loss: 38766.179688 Best loss: 1412.347290 Accuracy: 92.69% 4 Validation loss: 16240.206055 Best loss: 1412.347290 Accuracy: 95.70% 5 Validation loss: 18075.625000 Best loss: 1412.347290 Accuracy: 96.21% 6 Validation loss: 18478.841797 Best loss: 1412.347290 Accuracy: 94.76% 7 Validation loss: 11610.210938 Best loss: 1412.347290 Accuracy: 95.70% 8 Validation loss: 13099.498047 Best loss: 1412.347290 Accuracy: 90.97% 9 Validation loss: 10827.782227 Best loss: 1412.347290 Accuracy: 96.29% 10 Validation loss: 15759.868164 Best loss: 1412.347290 Accuracy: 96.68% 11 Validation loss: 291359104.000000 Best loss: 1412.347290 Accuracy: 93.67% 12 Validation loss: 16141494.000000 Best loss: 1412.347290 Accuracy: 95.90% 13 Validation loss: 9132939.000000 Best loss: 1412.347290 Accuracy: 96.72% 14 Validation loss: 4223947.000000 Best loss: 1412.347290 Accuracy: 96.33% 15 Validation loss: 4438993.000000 Best loss: 1412.347290 Accuracy: 96.01% 16 Validation loss: 5700826.000000 Best loss: 1412.347290 Accuracy: 96.64% 17 Validation loss: 3150014.500000 Best loss: 1412.347290 Accuracy: 96.33% 18 Validation loss: 3493170.250000 Best loss: 1412.347290 Accuracy: 96.13% 19 Validation loss: 2641357.250000 Best loss: 1412.347290 Accuracy: 96.44% 20 Validation loss: 2338019.500000 Best loss: 1412.347290 Accuracy: 96.29% 21 Validation loss: 2016419.875000 Best loss: 1412.347290 Accuracy: 95.70% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 28.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 59.494263 Best loss: 59.494263 Accuracy: 62.51% 1 Validation loss: 109.156296 Best loss: 59.494263 Accuracy: 76.51% 2 Validation loss: 244240.968750 Best loss: 59.494263 Accuracy: 56.41% 3 Validation loss: 132117.500000 Best loss: 59.494263 Accuracy: 51.09% 4 Validation loss: 104221.546875 Best loss: 59.494263 Accuracy: 51.95% 5 Validation loss: 66708.250000 Best loss: 59.494263 Accuracy: 59.50% 6 Validation loss: 32195.816406 Best loss: 59.494263 Accuracy: 82.17% 7 Validation loss: 43267.164062 Best loss: 59.494263 Accuracy: 87.06% 8 Validation loss: 22158.939453 Best loss: 59.494263 Accuracy: 89.05% 9 Validation loss: 13098.458984 Best loss: 59.494263 Accuracy: 83.31% 10 Validation loss: 14060.325195 Best loss: 59.494263 Accuracy: 93.63% 11 Validation loss: 9091641.000000 Best loss: 59.494263 Accuracy: 86.63% 12 Validation loss: 587287.875000 Best loss: 59.494263 Accuracy: 94.84% 13 Validation loss: 392257.906250 Best loss: 59.494263 Accuracy: 94.80% 14 Validation loss: 429001.718750 Best loss: 59.494263 Accuracy: 92.53% 15 Validation loss: 211936.593750 Best loss: 59.494263 Accuracy: 95.39% 16 Validation loss: 488304.875000 Best loss: 59.494263 Accuracy: 96.36% 17 Validation loss: 206532.734375 Best loss: 59.494263 Accuracy: 96.91% 18 Validation loss: 142195.515625 Best loss: 59.494263 Accuracy: 96.68% 19 Validation loss: 113703.429688 Best loss: 59.494263 Accuracy: 96.52% 20 Validation loss: 189319.875000 Best loss: 59.494263 Accuracy: 96.33% 21 Validation loss: 341914.500000 Best loss: 59.494263 Accuracy: 92.89% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 28.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 25634.882812 Best loss: 25634.882812 Accuracy: 38.66% 1 Validation loss: 2984.938232 Best loss: 2984.938232 Accuracy: 68.53% 2 Validation loss: 7394.522949 Best loss: 2984.938232 Accuracy: 54.89% 3 Validation loss: 4751.588867 Best loss: 2984.938232 Accuracy: 40.38% 4 Validation loss: 1587.797241 Best loss: 1587.797241 Accuracy: 64.07% 5 Validation loss: 414.736694 Best loss: 414.736694 Accuracy: 65.09% 6 Validation loss: 975.394043 Best loss: 414.736694 Accuracy: 71.97% 7 Validation loss: 162.628281 Best loss: 162.628281 Accuracy: 76.97% 8 Validation loss: 1885891.500000 Best loss: 162.628281 Accuracy: 84.71% 9 Validation loss: 568812.312500 Best loss: 162.628281 Accuracy: 86.67% 10 Validation loss: 407901.437500 Best loss: 162.628281 Accuracy: 89.91% 11 Validation loss: 117792.687500 Best loss: 162.628281 Accuracy: 93.35% 12 Validation loss: 167184.968750 Best loss: 162.628281 Accuracy: 91.48% 13 Validation loss: 81722.343750 Best loss: 162.628281 Accuracy: 95.58% 14 Validation loss: 121381.351562 Best loss: 162.628281 Accuracy: 93.55% 15 Validation loss: 79066.671875 Best loss: 162.628281 Accuracy: 95.43% 16 Validation loss: 94528.054688 Best loss: 162.628281 Accuracy: 93.94% 17 Validation loss: 48576.855469 Best loss: 162.628281 Accuracy: 95.58% 18 Validation loss: 60010.738281 Best loss: 162.628281 Accuracy: 95.23% 19 Validation loss: 45078.785156 Best loss: 162.628281 Accuracy: 95.04% 20 Validation loss: 60662.617188 Best loss: 162.628281 Accuracy: 95.70% 21 Validation loss: 69055.132812 Best loss: 162.628281 Accuracy: 95.74% 22 Validation loss: 31346.470703 Best loss: 162.628281 Accuracy: 96.44% 23 Validation loss: 32357.263672 Best loss: 162.628281 Accuracy: 94.37% 24 Validation loss: 46161.695312 Best loss: 162.628281 Accuracy: 94.80% 25 Validation loss: 29412.884766 Best loss: 162.628281 Accuracy: 96.79% 26 Validation loss: 34752.714844 Best loss: 162.628281 Accuracy: 95.97% 27 Validation loss: 2394721792.000000 Best loss: 162.628281 Accuracy: 50.63% 28 Validation loss: 6578249.000000 Best loss: 162.628281 Accuracy: 95.54% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 36.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 1.082493 Best loss: 1.082493 Accuracy: 51.37% 1 Validation loss: 1.332966 Best loss: 1.082493 Accuracy: 40.15% 2 Validation loss: 1.333849 Best loss: 1.082493 Accuracy: 42.85% 3 Validation loss: 1.611237 Best loss: 1.082493 Accuracy: 27.29% 4 Validation loss: 1.554624 Best loss: 1.082493 Accuracy: 26.04% 5 Validation loss: 1.448129 Best loss: 1.082493 Accuracy: 35.73% 6 Validation loss: 1.464206 Best loss: 1.082493 Accuracy: 30.49% 7 Validation loss: 1.427849 Best loss: 1.082493 Accuracy: 32.56% 8 Validation loss: 1.399452 Best loss: 1.082493 Accuracy: 35.42% 9 Validation loss: 1.702988 Best loss: 1.082493 Accuracy: 32.88% 10 Validation loss: 1.562043 Best loss: 1.082493 Accuracy: 27.21% 11 Validation loss: 3.230481 Best loss: 1.082493 Accuracy: 35.93% 12 Validation loss: 6.017581 Best loss: 1.082493 Accuracy: 36.36% 13 Validation loss: 5.356247 Best loss: 1.082493 Accuracy: 35.50% 14 Validation loss: 21.473650 Best loss: 1.082493 Accuracy: 22.01% 15 Validation loss: 9.638750 Best loss: 1.082493 Accuracy: 18.92% 16 Validation loss: 19.713840 Best loss: 1.082493 Accuracy: 20.29% 17 Validation loss: 34.239010 Best loss: 1.082493 Accuracy: 22.01% 18 Validation loss: 22.197912 Best loss: 1.082493 Accuracy: 21.11% 19 Validation loss: 132.702194 Best loss: 1.082493 Accuracy: 19.08% 20 Validation loss: 24.667154 Best loss: 1.082493 Accuracy: 29.44% 21 Validation loss: 86.340675 Best loss: 1.082493 Accuracy: 19.27% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6, total= 28.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 1.049644 Best loss: 1.049644 Accuracy: 52.85% 1 Validation loss: 0.955069 Best loss: 0.955069 Accuracy: 51.92% 2 Validation loss: 0.907248 Best loss: 0.907248 Accuracy: 56.57% 3 Validation loss: 1.453401 Best loss: 0.907248 Accuracy: 51.21% 4 Validation loss: 1.717936 Best loss: 0.907248 Accuracy: 22.87% 5 Validation loss: 1.591537 Best loss: 0.907248 Accuracy: 33.15% 6 Validation loss: 3.901171 Best loss: 0.907248 Accuracy: 19.27% 7 Validation loss: 5.843862 Best loss: 0.907248 Accuracy: 35.22% 8 Validation loss: 11.212295 Best loss: 0.907248 Accuracy: 34.48% 9 Validation loss: 38.244701 Best loss: 0.907248 Accuracy: 27.91% 10 Validation loss: 26.026497 Best loss: 0.907248 Accuracy: 19.27% 11 Validation loss: 11.646256 Best loss: 0.907248 Accuracy: 20.91% 12 Validation loss: 10.902725 Best loss: 0.907248 Accuracy: 21.38% 13 Validation loss: 6.401971 Best loss: 0.907248 Accuracy: 30.41% 14 Validation loss: 65.987854 Best loss: 0.907248 Accuracy: 22.24% 15 Validation loss: 65.878731 Best loss: 0.907248 Accuracy: 19.39% 16 Validation loss: 50.861473 Best loss: 0.907248 Accuracy: 21.11% 17 Validation loss: 32.517036 Best loss: 0.907248 Accuracy: 21.85% 18 Validation loss: 15.881490 Best loss: 0.907248 Accuracy: 25.76% 19 Validation loss: 20.802229 Best loss: 0.907248 Accuracy: 19.43% 20 Validation loss: 19.911308 Best loss: 0.907248 Accuracy: 22.52% 21 Validation loss: 109.773697 Best loss: 0.907248 Accuracy: 19.08% 22 Validation loss: 8.690555 Best loss: 0.907248 Accuracy: 27.37% 23 Validation loss: 7.647947 Best loss: 0.907248 Accuracy: 25.57% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6, total= 30.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6 0 Validation loss: 1.318410 Best loss: 1.318410 Accuracy: 34.83% 1 Validation loss: 1.061779 Best loss: 1.061779 Accuracy: 51.60% 2 Validation loss: 0.969370 Best loss: 0.969370 Accuracy: 54.42% 3 Validation loss: 1.166774 Best loss: 0.969370 Accuracy: 49.80% 4 Validation loss: 3.351317 Best loss: 0.969370 Accuracy: 30.02% 5 Validation loss: 2.707660 Best loss: 0.969370 Accuracy: 21.85% 6 Validation loss: 8.721770 Best loss: 0.969370 Accuracy: 19.08% 7 Validation loss: 15.820702 Best loss: 0.969370 Accuracy: 19.39% 8 Validation loss: 7.590757 Best loss: 0.969370 Accuracy: 35.93% 9 Validation loss: 6.723358 Best loss: 0.969370 Accuracy: 22.91% 10 Validation loss: 33.272354 Best loss: 0.969370 Accuracy: 18.84% 11 Validation loss: 14.994023 Best loss: 0.969370 Accuracy: 20.05% 12 Validation loss: 26.808008 Best loss: 0.969370 Accuracy: 20.91% 13 Validation loss: 131.838364 Best loss: 0.969370 Accuracy: 19.27% 14 Validation loss: 14.994564 Best loss: 0.969370 Accuracy: 35.30% 15 Validation loss: 6.307748 Best loss: 0.969370 Accuracy: 25.06% 16 Validation loss: 20.767559 Best loss: 0.969370 Accuracy: 36.12% 17 Validation loss: 22.837414 Best loss: 0.969370 Accuracy: 19.31% 18 Validation loss: 18.729441 Best loss: 0.969370 Accuracy: 35.18% 19 Validation loss: 51.215042 Best loss: 0.969370 Accuracy: 24.63% 20 Validation loss: 42.620411 Best loss: 0.969370 Accuracy: 23.38% 21 Validation loss: 13.419254 Best loss: 0.969370 Accuracy: 24.55% 22 Validation loss: 32.953999 Best loss: 0.969370 Accuracy: 20.91% 23 Validation loss: 229.458420 Best loss: 0.969370 Accuracy: 31.16% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=50, learning_rate=0.02, dropout_rate=0.6, total= 30.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.362992 Best loss: 0.362992 Accuracy: 90.85% 1 Validation loss: 0.494085 Best loss: 0.362992 Accuracy: 79.09% 2 Validation loss: 0.615718 Best loss: 0.362992 Accuracy: 72.67% 3 Validation loss: 0.811216 Best loss: 0.362992 Accuracy: 58.13% 4 Validation loss: 0.605926 Best loss: 0.362992 Accuracy: 72.24% 5 Validation loss: 1.014989 Best loss: 0.362992 Accuracy: 52.15% 6 Validation loss: 0.940183 Best loss: 0.362992 Accuracy: 52.07% 7 Validation loss: 0.982033 Best loss: 0.362992 Accuracy: 51.80% 8 Validation loss: 1.282226 Best loss: 0.362992 Accuracy: 45.35% 9 Validation loss: 1.304646 Best loss: 0.362992 Accuracy: 48.32% 10 Validation loss: 1.112928 Best loss: 0.362992 Accuracy: 53.95% 11 Validation loss: 1.207160 Best loss: 0.362992 Accuracy: 44.72% 12 Validation loss: 1.139664 Best loss: 0.362992 Accuracy: 52.81% 13 Validation loss: 1.203737 Best loss: 0.362992 Accuracy: 49.61% 14 Validation loss: 1.062539 Best loss: 0.362992 Accuracy: 53.09% 15 Validation loss: 1.453616 Best loss: 0.362992 Accuracy: 45.15% 16 Validation loss: 1.226611 Best loss: 0.362992 Accuracy: 52.97% 17 Validation loss: 1.166891 Best loss: 0.362992 Accuracy: 49.10% 18 Validation loss: 8.624493 Best loss: 0.362992 Accuracy: 20.91% 19 Validation loss: 3.195251 Best loss: 0.362992 Accuracy: 32.02% 20 Validation loss: 2.335213 Best loss: 0.362992 Accuracy: 39.80% 21 Validation loss: 3.404585 Best loss: 0.362992 Accuracy: 36.98% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.292257 Best loss: 0.292257 Accuracy: 90.70% 1 Validation loss: 0.445262 Best loss: 0.292257 Accuracy: 90.19% 2 Validation loss: 0.386447 Best loss: 0.292257 Accuracy: 88.94% 3 Validation loss: 0.603844 Best loss: 0.292257 Accuracy: 84.32% 4 Validation loss: 0.486474 Best loss: 0.292257 Accuracy: 79.67% 5 Validation loss: 1.036238 Best loss: 0.292257 Accuracy: 57.82% 6 Validation loss: 1.185621 Best loss: 0.292257 Accuracy: 52.97% 7 Validation loss: 0.914737 Best loss: 0.292257 Accuracy: 58.52% 8 Validation loss: 0.951357 Best loss: 0.292257 Accuracy: 55.24% 9 Validation loss: 1.044672 Best loss: 0.292257 Accuracy: 50.55% 10 Validation loss: 0.989678 Best loss: 0.292257 Accuracy: 54.96% 11 Validation loss: 0.988711 Best loss: 0.292257 Accuracy: 55.28% 12 Validation loss: 1.030556 Best loss: 0.292257 Accuracy: 52.11% 13 Validation loss: 0.920062 Best loss: 0.292257 Accuracy: 57.58% 14 Validation loss: 0.995311 Best loss: 0.292257 Accuracy: 55.59% 15 Validation loss: 0.971297 Best loss: 0.292257 Accuracy: 54.53% 16 Validation loss: 0.963072 Best loss: 0.292257 Accuracy: 53.52% 17 Validation loss: 1.109473 Best loss: 0.292257 Accuracy: 46.52% 18 Validation loss: 1.149400 Best loss: 0.292257 Accuracy: 48.36% 19 Validation loss: 1.787079 Best loss: 0.292257 Accuracy: 42.65% 20 Validation loss: 3.577943 Best loss: 0.292257 Accuracy: 45.19% 21 Validation loss: 1.309680 Best loss: 0.292257 Accuracy: 44.92% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4 0 Validation loss: 0.410611 Best loss: 0.410611 Accuracy: 90.11% 1 Validation loss: 0.717897 Best loss: 0.410611 Accuracy: 74.51% 2 Validation loss: 1.154367 Best loss: 0.410611 Accuracy: 55.67% 3 Validation loss: 1.526178 Best loss: 0.410611 Accuracy: 25.45% 4 Validation loss: 1.240086 Best loss: 0.410611 Accuracy: 39.99% 5 Validation loss: 1.267525 Best loss: 0.410611 Accuracy: 40.07% 6 Validation loss: 1.266617 Best loss: 0.410611 Accuracy: 39.91% 7 Validation loss: 1.138459 Best loss: 0.410611 Accuracy: 49.84% 8 Validation loss: 1.085570 Best loss: 0.410611 Accuracy: 52.03% 9 Validation loss: 1.346192 Best loss: 0.410611 Accuracy: 40.70% 10 Validation loss: 1.023459 Best loss: 0.410611 Accuracy: 54.14% 11 Validation loss: 1.047446 Best loss: 0.410611 Accuracy: 54.93% 12 Validation loss: 1.078008 Best loss: 0.410611 Accuracy: 50.27% 13 Validation loss: 0.980893 Best loss: 0.410611 Accuracy: 57.70% 14 Validation loss: 1.093766 Best loss: 0.410611 Accuracy: 50.47% 15 Validation loss: 1.049116 Best loss: 0.410611 Accuracy: 54.96% 16 Validation loss: 2.652541 Best loss: 0.410611 Accuracy: 46.13% 17 Validation loss: 3.351640 Best loss: 0.410611 Accuracy: 26.58% 18 Validation loss: 8.421195 Best loss: 0.410611 Accuracy: 23.69% 19 Validation loss: 1.475653 Best loss: 0.410611 Accuracy: 42.69% 20 Validation loss: 2.077420 Best loss: 0.410611 Accuracy: 44.25% 21 Validation loss: 1.376535 Best loss: 0.410611 Accuracy: 44.72% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.01, dropout_rate=0.4, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2 0 Validation loss: 0.270729 Best loss: 0.270729 Accuracy: 91.75% 1 Validation loss: 0.169494 Best loss: 0.169494 Accuracy: 94.76% 2 Validation loss: 0.184343 Best loss: 0.169494 Accuracy: 95.39% 3 Validation loss: 0.187448 Best loss: 0.169494 Accuracy: 94.72% 4 Validation loss: 0.169316 Best loss: 0.169316 Accuracy: 95.58% 5 Validation loss: 0.189564 Best loss: 0.169316 Accuracy: 95.04% 6 Validation loss: 0.612354 Best loss: 0.169316 Accuracy: 80.77% 7 Validation loss: 0.698604 Best loss: 0.169316 Accuracy: 78.69% 8 Validation loss: 1.399056 Best loss: 0.169316 Accuracy: 38.62% 9 Validation loss: 0.637985 Best loss: 0.169316 Accuracy: 77.44% 10 Validation loss: 0.558470 Best loss: 0.169316 Accuracy: 84.44% 11 Validation loss: 0.510443 Best loss: 0.169316 Accuracy: 84.13% 12 Validation loss: 0.804361 Best loss: 0.169316 Accuracy: 65.79% 13 Validation loss: 0.519471 Best loss: 0.169316 Accuracy: 83.07% 14 Validation loss: 0.388074 Best loss: 0.169316 Accuracy: 89.44% 15 Validation loss: 0.467186 Best loss: 0.169316 Accuracy: 85.97% 16 Validation loss: 0.457681 Best loss: 0.169316 Accuracy: 84.87% 17 Validation loss: 0.428425 Best loss: 0.169316 Accuracy: 86.00% 18 Validation loss: 0.384840 Best loss: 0.169316 Accuracy: 87.33% 19 Validation loss: 4.229437 Best loss: 0.169316 Accuracy: 22.05% 20 Validation loss: 140.369659 Best loss: 0.169316 Accuracy: 20.99% 21 Validation loss: 73.289124 Best loss: 0.169316 Accuracy: 19.27% 22 Validation loss: 16.881472 Best loss: 0.169316 Accuracy: 19.08% 23 Validation loss: 41.797764 Best loss: 0.169316 Accuracy: 20.91% 24 Validation loss: 369.266388 Best loss: 0.169316 Accuracy: 19.27% 25 Validation loss: 195.226028 Best loss: 0.169316 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2, total= 5.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2 0 Validation loss: 0.237630 Best loss: 0.237630 Accuracy: 93.04% 1 Validation loss: 0.182381 Best loss: 0.182381 Accuracy: 95.82% 2 Validation loss: 0.153109 Best loss: 0.153109 Accuracy: 95.31% 3 Validation loss: 0.163811 Best loss: 0.153109 Accuracy: 95.54% 4 Validation loss: 0.171230 Best loss: 0.153109 Accuracy: 95.78% 5 Validation loss: 0.497398 Best loss: 0.153109 Accuracy: 83.19% 6 Validation loss: 0.476509 Best loss: 0.153109 Accuracy: 83.31% 7 Validation loss: 0.364290 Best loss: 0.153109 Accuracy: 93.39% 8 Validation loss: 3.643413 Best loss: 0.153109 Accuracy: 38.55% 9 Validation loss: 2.259750 Best loss: 0.153109 Accuracy: 35.77% 10 Validation loss: 64.535141 Best loss: 0.153109 Accuracy: 60.79% 11 Validation loss: 40.530701 Best loss: 0.153109 Accuracy: 25.68% 12 Validation loss: 7.143611 Best loss: 0.153109 Accuracy: 33.66% 13 Validation loss: 30.622223 Best loss: 0.153109 Accuracy: 21.38% 14 Validation loss: 26.099052 Best loss: 0.153109 Accuracy: 39.56% 15 Validation loss: 12.422328 Best loss: 0.153109 Accuracy: 33.35% 16 Validation loss: 9.600165 Best loss: 0.153109 Accuracy: 28.89% 17 Validation loss: 9.351021 Best loss: 0.153109 Accuracy: 28.62% 18 Validation loss: 4.706139 Best loss: 0.153109 Accuracy: 41.44% 19 Validation loss: 3.049348 Best loss: 0.153109 Accuracy: 35.85% 20 Validation loss: 9.239480 Best loss: 0.153109 Accuracy: 32.33% 21 Validation loss: 5.381490 Best loss: 0.153109 Accuracy: 31.86% 22 Validation loss: 9.788671 Best loss: 0.153109 Accuracy: 30.77% 23 Validation loss: 3.246843 Best loss: 0.153109 Accuracy: 28.89% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2, total= 4.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2 0 Validation loss: 0.386873 Best loss: 0.386873 Accuracy: 87.14% 1 Validation loss: 0.160089 Best loss: 0.160089 Accuracy: 95.39% 2 Validation loss: 0.153577 Best loss: 0.153577 Accuracy: 95.31% 3 Validation loss: 0.152284 Best loss: 0.152284 Accuracy: 96.52% 4 Validation loss: 0.157765 Best loss: 0.152284 Accuracy: 95.78% 5 Validation loss: 0.220808 Best loss: 0.152284 Accuracy: 94.61% 6 Validation loss: 0.248290 Best loss: 0.152284 Accuracy: 92.49% 7 Validation loss: 0.340570 Best loss: 0.152284 Accuracy: 90.03% 8 Validation loss: 0.473125 Best loss: 0.152284 Accuracy: 82.64% 9 Validation loss: 0.514786 Best loss: 0.152284 Accuracy: 76.90% 10 Validation loss: 0.663446 Best loss: 0.152284 Accuracy: 74.43% 11 Validation loss: 0.906738 Best loss: 0.152284 Accuracy: 65.44% 12 Validation loss: 0.960411 Best loss: 0.152284 Accuracy: 56.72% 13 Validation loss: 0.986511 Best loss: 0.152284 Accuracy: 63.25% 14 Validation loss: 1.110791 Best loss: 0.152284 Accuracy: 58.52% 15 Validation loss: 0.659851 Best loss: 0.152284 Accuracy: 75.84% 16 Validation loss: 0.544529 Best loss: 0.152284 Accuracy: 84.36% 17 Validation loss: 0.387461 Best loss: 0.152284 Accuracy: 88.23% 18 Validation loss: 0.403538 Best loss: 0.152284 Accuracy: 88.51% 19 Validation loss: 0.447645 Best loss: 0.152284 Accuracy: 85.46% 20 Validation loss: 0.365876 Best loss: 0.152284 Accuracy: 88.62% 21 Validation loss: 0.348691 Best loss: 0.152284 Accuracy: 89.41% 22 Validation loss: 0.331883 Best loss: 0.152284 Accuracy: 89.87% 23 Validation loss: 0.666642 Best loss: 0.152284 Accuracy: 84.21% 24 Validation loss: 0.957908 Best loss: 0.152284 Accuracy: 57.74% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.05, dropout_rate=0.2, total= 5.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.158689 Best loss: 0.158689 Accuracy: 95.97% 1 Validation loss: 0.198062 Best loss: 0.158689 Accuracy: 95.50% 2 Validation loss: 0.148045 Best loss: 0.148045 Accuracy: 96.60% 3 Validation loss: 0.164509 Best loss: 0.148045 Accuracy: 95.82% 4 Validation loss: 0.134933 Best loss: 0.134933 Accuracy: 97.11% 5 Validation loss: 0.122227 Best loss: 0.122227 Accuracy: 96.79% 6 Validation loss: 0.120540 Best loss: 0.120540 Accuracy: 96.83% 7 Validation loss: 0.127748 Best loss: 0.120540 Accuracy: 96.95% 8 Validation loss: 0.150633 Best loss: 0.120540 Accuracy: 97.03% 9 Validation loss: 0.126395 Best loss: 0.120540 Accuracy: 96.68% 10 Validation loss: 0.161884 Best loss: 0.120540 Accuracy: 96.44% 11 Validation loss: 0.127621 Best loss: 0.120540 Accuracy: 97.03% 12 Validation loss: 0.099513 Best loss: 0.099513 Accuracy: 97.34% 13 Validation loss: 0.138953 Best loss: 0.099513 Accuracy: 96.87% 14 Validation loss: 0.149920 Best loss: 0.099513 Accuracy: 96.83% 15 Validation loss: 0.184802 Best loss: 0.099513 Accuracy: 96.21% 16 Validation loss: 0.140611 Best loss: 0.099513 Accuracy: 96.99% 17 Validation loss: 0.143267 Best loss: 0.099513 Accuracy: 96.83% 18 Validation loss: 0.154519 Best loss: 0.099513 Accuracy: 96.44% 19 Validation loss: 0.161822 Best loss: 0.099513 Accuracy: 96.91% 20 Validation loss: 0.138948 Best loss: 0.099513 Accuracy: 96.79% 21 Validation loss: 0.144114 Best loss: 0.099513 Accuracy: 96.52% 22 Validation loss: 0.148351 Best loss: 0.099513 Accuracy: 96.56% 23 Validation loss: 0.133223 Best loss: 0.099513 Accuracy: 97.19% 24 Validation loss: 0.152711 Best loss: 0.099513 Accuracy: 96.87% 25 Validation loss: 0.165292 Best loss: 0.099513 Accuracy: 96.40% 26 Validation loss: 0.167398 Best loss: 0.099513 Accuracy: 96.21% 27 Validation loss: 0.245352 Best loss: 0.099513 Accuracy: 90.66% 28 Validation loss: 0.167342 Best loss: 0.099513 Accuracy: 97.15% 29 Validation loss: 0.157624 Best loss: 0.099513 Accuracy: 97.15% 30 Validation loss: 0.149627 Best loss: 0.099513 Accuracy: 96.48% 31 Validation loss: 0.175124 Best loss: 0.099513 Accuracy: 96.33% 32 Validation loss: 0.167069 Best loss: 0.099513 Accuracy: 96.21% 33 Validation loss: 0.157370 Best loss: 0.099513 Accuracy: 96.56% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 38.2s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.150146 Best loss: 0.150146 Accuracy: 96.01% 1 Validation loss: 0.150088 Best loss: 0.150088 Accuracy: 96.21% 2 Validation loss: 0.138400 Best loss: 0.138400 Accuracy: 96.40% 3 Validation loss: 0.180562 Best loss: 0.138400 Accuracy: 95.31% 4 Validation loss: 0.118254 Best loss: 0.118254 Accuracy: 96.83% 5 Validation loss: 0.133439 Best loss: 0.118254 Accuracy: 96.95% 6 Validation loss: 0.116905 Best loss: 0.116905 Accuracy: 97.15% 7 Validation loss: 0.126425 Best loss: 0.116905 Accuracy: 96.44% 8 Validation loss: 0.132079 Best loss: 0.116905 Accuracy: 96.33% 9 Validation loss: 0.134266 Best loss: 0.116905 Accuracy: 96.79% 10 Validation loss: 0.136152 Best loss: 0.116905 Accuracy: 96.52% 11 Validation loss: 0.173089 Best loss: 0.116905 Accuracy: 95.15% 12 Validation loss: 0.133163 Best loss: 0.116905 Accuracy: 96.33% 13 Validation loss: 0.127470 Best loss: 0.116905 Accuracy: 96.56% 14 Validation loss: 0.150546 Best loss: 0.116905 Accuracy: 96.09% 15 Validation loss: 0.218844 Best loss: 0.116905 Accuracy: 95.86% 16 Validation loss: 0.158963 Best loss: 0.116905 Accuracy: 96.09% 17 Validation loss: 0.139062 Best loss: 0.116905 Accuracy: 96.87% 18 Validation loss: 0.154833 Best loss: 0.116905 Accuracy: 96.52% 19 Validation loss: 0.159713 Best loss: 0.116905 Accuracy: 95.97% 20 Validation loss: 0.135768 Best loss: 0.116905 Accuracy: 96.25% 21 Validation loss: 0.281000 Best loss: 0.116905 Accuracy: 96.72% 22 Validation loss: 0.152121 Best loss: 0.116905 Accuracy: 96.25% 23 Validation loss: 0.138280 Best loss: 0.116905 Accuracy: 95.97% 24 Validation loss: 0.166503 Best loss: 0.116905 Accuracy: 96.48% 25 Validation loss: 0.196229 Best loss: 0.116905 Accuracy: 95.66% 26 Validation loss: 0.131469 Best loss: 0.116905 Accuracy: 96.21% 27 Validation loss: 0.150554 Best loss: 0.116905 Accuracy: 96.09% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 31.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.163418 Best loss: 0.163418 Accuracy: 95.62% 1 Validation loss: 0.144450 Best loss: 0.144450 Accuracy: 96.52% 2 Validation loss: 0.152336 Best loss: 0.144450 Accuracy: 96.25% 3 Validation loss: 0.130403 Best loss: 0.130403 Accuracy: 96.52% 4 Validation loss: 0.163374 Best loss: 0.130403 Accuracy: 96.40% 5 Validation loss: 0.137815 Best loss: 0.130403 Accuracy: 96.25% 6 Validation loss: 0.123722 Best loss: 0.123722 Accuracy: 96.83% 7 Validation loss: 0.136258 Best loss: 0.123722 Accuracy: 96.99% 8 Validation loss: 0.127861 Best loss: 0.123722 Accuracy: 97.11% 9 Validation loss: 0.193102 Best loss: 0.123722 Accuracy: 95.58% 10 Validation loss: 0.139400 Best loss: 0.123722 Accuracy: 96.48% 11 Validation loss: 0.173675 Best loss: 0.123722 Accuracy: 96.72% 12 Validation loss: 0.138151 Best loss: 0.123722 Accuracy: 96.83% 13 Validation loss: 0.162686 Best loss: 0.123722 Accuracy: 95.93% 14 Validation loss: 0.129486 Best loss: 0.123722 Accuracy: 96.72% 15 Validation loss: 0.180377 Best loss: 0.123722 Accuracy: 95.78% 16 Validation loss: 0.176633 Best loss: 0.123722 Accuracy: 96.72% 17 Validation loss: 0.164958 Best loss: 0.123722 Accuracy: 96.17% 18 Validation loss: 0.165420 Best loss: 0.123722 Accuracy: 96.48% 19 Validation loss: 0.207036 Best loss: 0.123722 Accuracy: 95.82% 20 Validation loss: 0.155394 Best loss: 0.123722 Accuracy: 97.11% 21 Validation loss: 0.184615 Best loss: 0.123722 Accuracy: 96.40% 22 Validation loss: 0.155347 Best loss: 0.123722 Accuracy: 96.29% 23 Validation loss: 0.217655 Best loss: 0.123722 Accuracy: 95.19% 24 Validation loss: 0.186493 Best loss: 0.123722 Accuracy: 96.33% 25 Validation loss: 0.153818 Best loss: 0.123722 Accuracy: 96.68% 26 Validation loss: 0.148489 Best loss: 0.123722 Accuracy: 96.87% 27 Validation loss: 0.177487 Best loss: 0.123722 Accuracy: 96.33% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 31.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 18.128426 Best loss: 18.128426 Accuracy: 31.59% 1 Validation loss: 331.000641 Best loss: 18.128426 Accuracy: 18.73% 2 Validation loss: 109.788803 Best loss: 18.128426 Accuracy: 22.01% 3 Validation loss: 275.842072 Best loss: 18.128426 Accuracy: 18.73% 4 Validation loss: 179.647934 Best loss: 18.128426 Accuracy: 22.01% 5 Validation loss: 572.269043 Best loss: 18.128426 Accuracy: 19.27% 6 Validation loss: 432.399200 Best loss: 18.128426 Accuracy: 23.46% 7 Validation loss: 144.376007 Best loss: 18.128426 Accuracy: 19.70% 8 Validation loss: 149.743668 Best loss: 18.128426 Accuracy: 33.58% 9 Validation loss: 581.220459 Best loss: 18.128426 Accuracy: 18.73% 10 Validation loss: 113.493431 Best loss: 18.128426 Accuracy: 23.42% 11 Validation loss: 81.770050 Best loss: 18.128426 Accuracy: 23.96% 12 Validation loss: 127.837738 Best loss: 18.128426 Accuracy: 19.27% 13 Validation loss: 53.214199 Best loss: 18.128426 Accuracy: 28.54% 14 Validation loss: 347.741089 Best loss: 18.128426 Accuracy: 19.08% 15 Validation loss: 94.481544 Best loss: 18.128426 Accuracy: 35.22% 16 Validation loss: 664.470337 Best loss: 18.128426 Accuracy: 19.27% 17 Validation loss: 210.466949 Best loss: 18.128426 Accuracy: 30.65% 18 Validation loss: 202.436401 Best loss: 18.128426 Accuracy: 21.11% 19 Validation loss: 444.651031 Best loss: 18.128426 Accuracy: 18.73% 20 Validation loss: 206.614487 Best loss: 18.128426 Accuracy: 33.07% 21 Validation loss: 114.186508 Best loss: 18.128426 Accuracy: 24.43% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5, total= 29.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 43.879890 Best loss: 43.879890 Accuracy: 22.32% 1 Validation loss: 61.687553 Best loss: 43.879890 Accuracy: 18.73% 2 Validation loss: 967.279541 Best loss: 43.879890 Accuracy: 18.73% 3 Validation loss: 2913.138184 Best loss: 43.879890 Accuracy: 19.08% 4 Validation loss: 2461.689453 Best loss: 43.879890 Accuracy: 20.91% 5 Validation loss: 270.576569 Best loss: 43.879890 Accuracy: 19.08% 6 Validation loss: 1574.965698 Best loss: 43.879890 Accuracy: 18.73% 7 Validation loss: 493.425568 Best loss: 43.879890 Accuracy: 19.08% 8 Validation loss: 1206.371582 Best loss: 43.879890 Accuracy: 18.73% 9 Validation loss: 858.298767 Best loss: 43.879890 Accuracy: 18.73% 10 Validation loss: 674.170105 Best loss: 43.879890 Accuracy: 22.01% 11 Validation loss: 1406.147827 Best loss: 43.879890 Accuracy: 20.91% 12 Validation loss: 2748.497559 Best loss: 43.879890 Accuracy: 20.91% 13 Validation loss: 936.622070 Best loss: 43.879890 Accuracy: 22.01% 14 Validation loss: 13589.852539 Best loss: 43.879890 Accuracy: 18.73% 15 Validation loss: 7084.933594 Best loss: 43.879890 Accuracy: 18.73% 16 Validation loss: 2412.103271 Best loss: 43.879890 Accuracy: 19.08% 17 Validation loss: 4063.729980 Best loss: 43.879890 Accuracy: 19.08% 18 Validation loss: 8401.020508 Best loss: 43.879890 Accuracy: 18.73% 19 Validation loss: 2386.400635 Best loss: 43.879890 Accuracy: 22.01% 20 Validation loss: 6534.888184 Best loss: 43.879890 Accuracy: 22.01% 21 Validation loss: 5373.378906 Best loss: 43.879890 Accuracy: 19.27% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5, total= 28.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 20.267097 Best loss: 20.267097 Accuracy: 19.08% 1 Validation loss: 32.518642 Best loss: 20.267097 Accuracy: 19.27% 2 Validation loss: 38.899395 Best loss: 20.267097 Accuracy: 19.27% 3 Validation loss: 230.750412 Best loss: 20.267097 Accuracy: 19.08% 4 Validation loss: 1225.213135 Best loss: 20.267097 Accuracy: 19.27% 5 Validation loss: 1516.693115 Best loss: 20.267097 Accuracy: 19.27% 6 Validation loss: 137.910233 Best loss: 20.267097 Accuracy: 34.13% 7 Validation loss: 526.089355 Best loss: 20.267097 Accuracy: 18.73% 8 Validation loss: 1337.650269 Best loss: 20.267097 Accuracy: 18.73% 9 Validation loss: 1617.061646 Best loss: 20.267097 Accuracy: 19.27% 10 Validation loss: 479.757629 Best loss: 20.267097 Accuracy: 18.73% 11 Validation loss: 373.757690 Best loss: 20.267097 Accuracy: 18.84% 12 Validation loss: 755.925964 Best loss: 20.267097 Accuracy: 19.27% 13 Validation loss: 1679.173584 Best loss: 20.267097 Accuracy: 18.73% 14 Validation loss: 420.400604 Best loss: 20.267097 Accuracy: 21.19% 15 Validation loss: 1044.625244 Best loss: 20.267097 Accuracy: 18.73% 16 Validation loss: 7648.220703 Best loss: 20.267097 Accuracy: 22.01% 17 Validation loss: 2661.745361 Best loss: 20.267097 Accuracy: 18.73% 18 Validation loss: 25527.275391 Best loss: 20.267097 Accuracy: 22.01% 19 Validation loss: 12061.680664 Best loss: 20.267097 Accuracy: 22.01% 20 Validation loss: 5329.979980 Best loss: 20.267097 Accuracy: 22.01% 21 Validation loss: 11073.992188 Best loss: 20.267097 Accuracy: 18.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=50, learning_rate=0.05, dropout_rate=0.5, total= 28.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.129945 Best loss: 0.129945 Accuracy: 96.72% 1 Validation loss: 0.115178 Best loss: 0.115178 Accuracy: 97.22% 2 Validation loss: 0.134977 Best loss: 0.115178 Accuracy: 96.95% 3 Validation loss: 0.144689 Best loss: 0.115178 Accuracy: 97.03% 4 Validation loss: 0.146947 Best loss: 0.115178 Accuracy: 96.99% 5 Validation loss: 0.106505 Best loss: 0.106505 Accuracy: 97.46% 6 Validation loss: 0.121581 Best loss: 0.106505 Accuracy: 96.99% 7 Validation loss: 0.133286 Best loss: 0.106505 Accuracy: 96.52% 8 Validation loss: 0.149533 Best loss: 0.106505 Accuracy: 96.87% 9 Validation loss: 0.130585 Best loss: 0.106505 Accuracy: 95.97% 10 Validation loss: 0.142635 Best loss: 0.106505 Accuracy: 96.01% 11 Validation loss: 0.146291 Best loss: 0.106505 Accuracy: 96.48% 12 Validation loss: 0.135675 Best loss: 0.106505 Accuracy: 96.25% 13 Validation loss: 0.146023 Best loss: 0.106505 Accuracy: 96.48% 14 Validation loss: 0.138926 Best loss: 0.106505 Accuracy: 96.09% 15 Validation loss: 0.175925 Best loss: 0.106505 Accuracy: 96.87% 16 Validation loss: 0.136155 Best loss: 0.106505 Accuracy: 96.68% 17 Validation loss: 0.132038 Best loss: 0.106505 Accuracy: 97.03% 18 Validation loss: 0.143126 Best loss: 0.106505 Accuracy: 96.17% 19 Validation loss: 0.131280 Best loss: 0.106505 Accuracy: 96.68% 20 Validation loss: 0.131237 Best loss: 0.106505 Accuracy: 96.21% 21 Validation loss: 0.129269 Best loss: 0.106505 Accuracy: 96.76% 22 Validation loss: 0.140396 Best loss: 0.106505 Accuracy: 96.60% 23 Validation loss: 0.115269 Best loss: 0.106505 Accuracy: 97.11% 24 Validation loss: 0.144983 Best loss: 0.106505 Accuracy: 96.40% 25 Validation loss: 0.134564 Best loss: 0.106505 Accuracy: 96.33% 26 Validation loss: 0.128805 Best loss: 0.106505 Accuracy: 96.60% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 18.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.194581 Best loss: 0.194581 Accuracy: 94.96% 1 Validation loss: 0.136594 Best loss: 0.136594 Accuracy: 96.72% 2 Validation loss: 0.127996 Best loss: 0.127996 Accuracy: 96.91% 3 Validation loss: 0.122071 Best loss: 0.122071 Accuracy: 96.64% 4 Validation loss: 0.114909 Best loss: 0.114909 Accuracy: 96.91% 5 Validation loss: 0.117737 Best loss: 0.114909 Accuracy: 97.07% 6 Validation loss: 0.117180 Best loss: 0.114909 Accuracy: 97.34% 7 Validation loss: 0.103825 Best loss: 0.103825 Accuracy: 97.38% 8 Validation loss: 0.126811 Best loss: 0.103825 Accuracy: 96.79% 9 Validation loss: 0.121432 Best loss: 0.103825 Accuracy: 96.87% 10 Validation loss: 0.125171 Best loss: 0.103825 Accuracy: 97.22% 11 Validation loss: 0.117788 Best loss: 0.103825 Accuracy: 96.79% 12 Validation loss: 0.124568 Best loss: 0.103825 Accuracy: 95.82% 13 Validation loss: 0.140443 Best loss: 0.103825 Accuracy: 96.17% 14 Validation loss: 0.117477 Best loss: 0.103825 Accuracy: 96.36% 15 Validation loss: 0.134944 Best loss: 0.103825 Accuracy: 96.60% 16 Validation loss: 0.112049 Best loss: 0.103825 Accuracy: 96.76% 17 Validation loss: 0.155668 Best loss: 0.103825 Accuracy: 94.57% 18 Validation loss: 0.156786 Best loss: 0.103825 Accuracy: 95.47% 19 Validation loss: 0.140099 Best loss: 0.103825 Accuracy: 96.13% 20 Validation loss: 0.130391 Best loss: 0.103825 Accuracy: 96.33% 21 Validation loss: 0.128904 Best loss: 0.103825 Accuracy: 96.25% 22 Validation loss: 0.140907 Best loss: 0.103825 Accuracy: 96.21% 23 Validation loss: 0.148804 Best loss: 0.103825 Accuracy: 95.97% 24 Validation loss: 0.135958 Best loss: 0.103825 Accuracy: 95.70% 25 Validation loss: 0.175220 Best loss: 0.103825 Accuracy: 94.96% 26 Validation loss: 0.189956 Best loss: 0.103825 Accuracy: 95.47% 27 Validation loss: 0.136238 Best loss: 0.103825 Accuracy: 95.97% 28 Validation loss: 0.163376 Best loss: 0.103825 Accuracy: 95.66% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 20.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.148877 Best loss: 0.148877 Accuracy: 96.05% 1 Validation loss: 0.142410 Best loss: 0.142410 Accuracy: 96.79% 2 Validation loss: 0.147049 Best loss: 0.142410 Accuracy: 96.56% 3 Validation loss: 0.129807 Best loss: 0.129807 Accuracy: 96.91% 4 Validation loss: 0.116429 Best loss: 0.116429 Accuracy: 96.91% 5 Validation loss: 0.133244 Best loss: 0.116429 Accuracy: 96.64% 6 Validation loss: 0.142279 Best loss: 0.116429 Accuracy: 96.72% 7 Validation loss: 0.131436 Best loss: 0.116429 Accuracy: 96.60% 8 Validation loss: 0.115137 Best loss: 0.115137 Accuracy: 96.95% 9 Validation loss: 0.119396 Best loss: 0.115137 Accuracy: 96.33% 10 Validation loss: 0.134392 Best loss: 0.115137 Accuracy: 95.66% 11 Validation loss: 0.128765 Best loss: 0.115137 Accuracy: 96.25% 12 Validation loss: 0.141436 Best loss: 0.115137 Accuracy: 94.72% 13 Validation loss: 0.153804 Best loss: 0.115137 Accuracy: 94.61% 14 Validation loss: 0.160147 Best loss: 0.115137 Accuracy: 95.58% 15 Validation loss: 0.148215 Best loss: 0.115137 Accuracy: 95.19% 16 Validation loss: 0.184125 Best loss: 0.115137 Accuracy: 95.66% 17 Validation loss: 0.170499 Best loss: 0.115137 Accuracy: 95.39% 18 Validation loss: 0.176585 Best loss: 0.115137 Accuracy: 94.21% 19 Validation loss: 0.171496 Best loss: 0.115137 Accuracy: 95.90% 20 Validation loss: 0.151949 Best loss: 0.115137 Accuracy: 95.78% 21 Validation loss: 0.157915 Best loss: 0.115137 Accuracy: 95.93% 22 Validation loss: 0.143276 Best loss: 0.115137 Accuracy: 95.50% 23 Validation loss: 0.163645 Best loss: 0.115137 Accuracy: 95.31% 24 Validation loss: 0.153556 Best loss: 0.115137 Accuracy: 95.86% 25 Validation loss: 0.153315 Best loss: 0.115137 Accuracy: 95.27% 26 Validation loss: 0.149439 Best loss: 0.115137 Accuracy: 96.09% 27 Validation loss: 0.138329 Best loss: 0.115137 Accuracy: 96.56% 28 Validation loss: 0.155754 Best loss: 0.115137 Accuracy: 95.62% 29 Validation loss: 0.165014 Best loss: 0.115137 Accuracy: 94.61% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=120, batch_size=100, learning_rate=0.01, dropout_rate=0.5, total= 20.7s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6 0 Validation loss: 1.614762 Best loss: 1.614762 Accuracy: 19.27% 1 Validation loss: 1.704116 Best loss: 1.614762 Accuracy: 22.01% 2 Validation loss: 1.622210 Best loss: 1.614762 Accuracy: 20.91% 3 Validation loss: 1.656982 Best loss: 1.614762 Accuracy: 19.08% 4 Validation loss: 1.710966 Best loss: 1.614762 Accuracy: 19.27% 5 Validation loss: 1.639604 Best loss: 1.614762 Accuracy: 18.73% 6 Validation loss: 1.670872 Best loss: 1.614762 Accuracy: 19.08% 7 Validation loss: 1.617985 Best loss: 1.614762 Accuracy: 22.01% 8 Validation loss: 1.657077 Best loss: 1.614762 Accuracy: 22.01% 9 Validation loss: 1.706992 Best loss: 1.614762 Accuracy: 22.01% 10 Validation loss: 1.645335 Best loss: 1.614762 Accuracy: 19.08% 11 Validation loss: 1.686759 Best loss: 1.614762 Accuracy: 22.01% 12 Validation loss: 1.819941 Best loss: 1.614762 Accuracy: 22.01% 13 Validation loss: 1.637530 Best loss: 1.614762 Accuracy: 22.01% 14 Validation loss: 1.715499 Best loss: 1.614762 Accuracy: 18.73% 15 Validation loss: 1.629649 Best loss: 1.614762 Accuracy: 22.01% 16 Validation loss: 1.677383 Best loss: 1.614762 Accuracy: 19.27% 17 Validation loss: 1.647645 Best loss: 1.614762 Accuracy: 19.08% 18 Validation loss: 1.711800 Best loss: 1.614762 Accuracy: 18.73% 19 Validation loss: 1.658469 Best loss: 1.614762 Accuracy: 22.01% 20 Validation loss: 1.654119 Best loss: 1.614762 Accuracy: 20.91% 21 Validation loss: 1.614728 Best loss: 1.614728 Accuracy: 19.27% 22 Validation loss: 1.674412 Best loss: 1.614728 Accuracy: 18.73% 23 Validation loss: 1.655149 Best loss: 1.614728 Accuracy: 22.01% 24 Validation loss: 1.716197 Best loss: 1.614728 Accuracy: 22.01% 25 Validation loss: 1.685928 Best loss: 1.614728 Accuracy: 22.01% 26 Validation loss: 1.666684 Best loss: 1.614728 Accuracy: 19.08% 27 Validation loss: 1.646759 Best loss: 1.614728 Accuracy: 18.73% 28 Validation loss: 1.652355 Best loss: 1.614728 Accuracy: 19.08% 29 Validation loss: 1.686789 Best loss: 1.614728 Accuracy: 19.27% 30 Validation loss: 1.675710 Best loss: 1.614728 Accuracy: 19.08% 31 Validation loss: 1.681892 Best loss: 1.614728 Accuracy: 22.01% 32 Validation loss: 1.692425 Best loss: 1.614728 Accuracy: 19.08% 33 Validation loss: 1.751532 Best loss: 1.614728 Accuracy: 19.08% 34 Validation loss: 1.654519 Best loss: 1.614728 Accuracy: 22.01% 35 Validation loss: 1.630639 Best loss: 1.614728 Accuracy: 19.08% 36 Validation loss: 1.650976 Best loss: 1.614728 Accuracy: 20.91% 37 Validation loss: 1.653056 Best loss: 1.614728 Accuracy: 18.73% 38 Validation loss: 1.751480 Best loss: 1.614728 Accuracy: 19.27% 39 Validation loss: 1.630422 Best loss: 1.614728 Accuracy: 20.91% 40 Validation loss: 1.767942 Best loss: 1.614728 Accuracy: 22.01% 41 Validation loss: 1.681653 Best loss: 1.614728 Accuracy: 19.27% 42 Validation loss: 1.654741 Best loss: 1.614728 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6, total= 7.3s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6 0 Validation loss: 1.614336 Best loss: 1.614336 Accuracy: 22.01% 1 Validation loss: 1.663885 Best loss: 1.614336 Accuracy: 19.27% 2 Validation loss: 1.641195 Best loss: 1.614336 Accuracy: 19.08% 3 Validation loss: 1.649371 Best loss: 1.614336 Accuracy: 18.73% 4 Validation loss: 1.641371 Best loss: 1.614336 Accuracy: 19.27% 5 Validation loss: 1.643060 Best loss: 1.614336 Accuracy: 20.91% 6 Validation loss: 1.750494 Best loss: 1.614336 Accuracy: 19.08% 7 Validation loss: 1.675419 Best loss: 1.614336 Accuracy: 19.08% 8 Validation loss: 1.751396 Best loss: 1.614336 Accuracy: 22.01% 9 Validation loss: 1.763243 Best loss: 1.614336 Accuracy: 19.27% 10 Validation loss: 1.642440 Best loss: 1.614336 Accuracy: 22.01% 11 Validation loss: 1.682216 Best loss: 1.614336 Accuracy: 22.01% 12 Validation loss: 1.644665 Best loss: 1.614336 Accuracy: 19.08% 13 Validation loss: 1.689191 Best loss: 1.614336 Accuracy: 19.08% 14 Validation loss: 1.761742 Best loss: 1.614336 Accuracy: 22.01% 15 Validation loss: 1.765535 Best loss: 1.614336 Accuracy: 22.01% 16 Validation loss: 1.659895 Best loss: 1.614336 Accuracy: 19.27% 17 Validation loss: 1.655919 Best loss: 1.614336 Accuracy: 18.73% 18 Validation loss: 1.620253 Best loss: 1.614336 Accuracy: 22.01% 19 Validation loss: 1.642314 Best loss: 1.614336 Accuracy: 22.01% 20 Validation loss: 1.641569 Best loss: 1.614336 Accuracy: 20.91% 21 Validation loss: 1.788227 Best loss: 1.614336 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6, total= 4.0s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6 0 Validation loss: 1.630719 Best loss: 1.630719 Accuracy: 19.08% 1 Validation loss: 1.615110 Best loss: 1.615110 Accuracy: 19.27% 2 Validation loss: 1.702417 Best loss: 1.615110 Accuracy: 19.27% 3 Validation loss: 1.702781 Best loss: 1.615110 Accuracy: 18.73% 4 Validation loss: 1.654711 Best loss: 1.615110 Accuracy: 20.91% 5 Validation loss: 1.618184 Best loss: 1.615110 Accuracy: 20.91% 6 Validation loss: 1.651560 Best loss: 1.615110 Accuracy: 22.01% 7 Validation loss: 1.615516 Best loss: 1.615110 Accuracy: 18.73% 8 Validation loss: 1.704277 Best loss: 1.615110 Accuracy: 22.01% 9 Validation loss: 1.648605 Best loss: 1.615110 Accuracy: 19.27% 10 Validation loss: 1.673079 Best loss: 1.615110 Accuracy: 22.01% 11 Validation loss: 1.654919 Best loss: 1.615110 Accuracy: 22.01% 12 Validation loss: 1.619806 Best loss: 1.615110 Accuracy: 22.01% 13 Validation loss: 1.761420 Best loss: 1.615110 Accuracy: 19.27% 14 Validation loss: 1.879732 Best loss: 1.615110 Accuracy: 20.91% 15 Validation loss: 1.640172 Best loss: 1.615110 Accuracy: 22.01% 16 Validation loss: 1.700122 Best loss: 1.615110 Accuracy: 18.73% 17 Validation loss: 1.679209 Best loss: 1.615110 Accuracy: 19.08% 18 Validation loss: 1.634598 Best loss: 1.615110 Accuracy: 19.27% 19 Validation loss: 1.711810 Best loss: 1.615110 Accuracy: 22.01% 20 Validation loss: 1.636249 Best loss: 1.615110 Accuracy: 19.08% 21 Validation loss: 1.734940 Best loss: 1.615110 Accuracy: 22.01% 22 Validation loss: 1.623995 Best loss: 1.615110 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.6, total= 4.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 2.649235 Best loss: 2.649235 Accuracy: 20.99% 1 Validation loss: 3.205683 Best loss: 2.649235 Accuracy: 25.29% 2 Validation loss: 315.591370 Best loss: 2.649235 Accuracy: 19.12% 3 Validation loss: 92.484886 Best loss: 2.649235 Accuracy: 19.51% 4 Validation loss: 56.131119 Best loss: 2.649235 Accuracy: 34.13% 5 Validation loss: 69.267570 Best loss: 2.649235 Accuracy: 22.01% 6 Validation loss: 445.103058 Best loss: 2.649235 Accuracy: 18.80% 7 Validation loss: 511.625885 Best loss: 2.649235 Accuracy: 40.23% 8 Validation loss: 174.062225 Best loss: 2.649235 Accuracy: 19.08% 9 Validation loss: 72.774460 Best loss: 2.649235 Accuracy: 36.79% 10 Validation loss: 1502.804321 Best loss: 2.649235 Accuracy: 19.27% 11 Validation loss: 279.140930 Best loss: 2.649235 Accuracy: 35.34% 12 Validation loss: 177.715103 Best loss: 2.649235 Accuracy: 18.84% 13 Validation loss: 309.504028 Best loss: 2.649235 Accuracy: 22.01% 14 Validation loss: 129.038376 Best loss: 2.649235 Accuracy: 25.33% 15 Validation loss: 195.906219 Best loss: 2.649235 Accuracy: 24.08% 16 Validation loss: 64.864395 Best loss: 2.649235 Accuracy: 26.54% 17 Validation loss: 129.614563 Best loss: 2.649235 Accuracy: 19.08% 18 Validation loss: 105.575890 Best loss: 2.649235 Accuracy: 20.05% 19 Validation loss: 3958.864014 Best loss: 2.649235 Accuracy: 22.01% 20 Validation loss: 451.940613 Best loss: 2.649235 Accuracy: 20.91% 21 Validation loss: 957.005249 Best loss: 2.649235 Accuracy: 19.66% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 123.693314 Best loss: 123.693314 Accuracy: 20.91% 1 Validation loss: 118.254349 Best loss: 118.254349 Accuracy: 29.52% 2 Validation loss: 91.055183 Best loss: 91.055183 Accuracy: 18.73% 3 Validation loss: 208.868057 Best loss: 91.055183 Accuracy: 22.01% 4 Validation loss: 272.513672 Best loss: 91.055183 Accuracy: 28.38% 5 Validation loss: 197.962280 Best loss: 91.055183 Accuracy: 19.55% 6 Validation loss: 204.944580 Best loss: 91.055183 Accuracy: 19.27% 7 Validation loss: 233.907501 Best loss: 91.055183 Accuracy: 19.19% 8 Validation loss: 41.115105 Best loss: 41.115105 Accuracy: 32.29% 9 Validation loss: 148.551926 Best loss: 41.115105 Accuracy: 22.60% 10 Validation loss: 224.838272 Best loss: 41.115105 Accuracy: 22.01% 11 Validation loss: 143.100662 Best loss: 41.115105 Accuracy: 24.71% 12 Validation loss: 248.561127 Best loss: 41.115105 Accuracy: 20.91% 13 Validation loss: 213.957687 Best loss: 41.115105 Accuracy: 27.56% 14 Validation loss: 567.785889 Best loss: 41.115105 Accuracy: 18.96% 15 Validation loss: 359.554962 Best loss: 41.115105 Accuracy: 19.08% 16 Validation loss: 1183.377686 Best loss: 41.115105 Accuracy: 20.91% 17 Validation loss: 595.186401 Best loss: 41.115105 Accuracy: 19.08% 18 Validation loss: 408.244843 Best loss: 41.115105 Accuracy: 19.19% 19 Validation loss: 257.052399 Best loss: 41.115105 Accuracy: 20.29% 20 Validation loss: 1921.651245 Best loss: 41.115105 Accuracy: 19.08% 21 Validation loss: 494.277405 Best loss: 41.115105 Accuracy: 20.29% 22 Validation loss: 219.515076 Best loss: 41.115105 Accuracy: 20.84% 23 Validation loss: 606.455505 Best loss: 41.115105 Accuracy: 19.12% 24 Validation loss: 1334.113037 Best loss: 41.115105 Accuracy: 19.04% 25 Validation loss: 915.216797 Best loss: 41.115105 Accuracy: 20.91% 26 Validation loss: 228.503906 Best loss: 41.115105 Accuracy: 36.20% 27 Validation loss: 1306.983521 Best loss: 41.115105 Accuracy: 19.90% 28 Validation loss: 266.959503 Best loss: 41.115105 Accuracy: 18.76% 29 Validation loss: 605.031494 Best loss: 41.115105 Accuracy: 22.67% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 3.0min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.411339 Best loss: 1.411339 Accuracy: 37.49% 1 Validation loss: 1.539100 Best loss: 1.411339 Accuracy: 30.02% 2 Validation loss: 177.192978 Best loss: 1.411339 Accuracy: 19.35% 3 Validation loss: 595.369568 Best loss: 1.411339 Accuracy: 20.91% 4 Validation loss: 134.018036 Best loss: 1.411339 Accuracy: 18.73% 5 Validation loss: 154.289490 Best loss: 1.411339 Accuracy: 18.73% 6 Validation loss: 113.942703 Best loss: 1.411339 Accuracy: 19.08% 7 Validation loss: 338.211334 Best loss: 1.411339 Accuracy: 19.08% 8 Validation loss: 679.186279 Best loss: 1.411339 Accuracy: 19.08% 9 Validation loss: 510.228943 Best loss: 1.411339 Accuracy: 19.27% 10 Validation loss: 334.092194 Best loss: 1.411339 Accuracy: 19.08% 11 Validation loss: 463.730499 Best loss: 1.411339 Accuracy: 19.08% 12 Validation loss: 396.077606 Best loss: 1.411339 Accuracy: 20.91% 13 Validation loss: 555.491516 Best loss: 1.411339 Accuracy: 22.01% 14 Validation loss: 529.587158 Best loss: 1.411339 Accuracy: 23.81% 15 Validation loss: 987.028564 Best loss: 1.411339 Accuracy: 20.91% 16 Validation loss: 99.043167 Best loss: 1.411339 Accuracy: 32.53% 17 Validation loss: 956.013123 Best loss: 1.411339 Accuracy: 19.08% 18 Validation loss: 423.170349 Best loss: 1.411339 Accuracy: 24.24% 19 Validation loss: 505.328186 Best loss: 1.411339 Accuracy: 19.27% 20 Validation loss: 580.130615 Best loss: 1.411339 Accuracy: 18.80% 21 Validation loss: 721.772888 Best loss: 1.411339 Accuracy: 26.04% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 1.854378 Best loss: 1.854378 Accuracy: 67.71% 1 Validation loss: 0.603591 Best loss: 0.603591 Accuracy: 89.13% 2 Validation loss: 12614.927734 Best loss: 0.603591 Accuracy: 92.38% 3 Validation loss: 10065.890625 Best loss: 0.603591 Accuracy: 82.56% 4 Validation loss: 2999.156738 Best loss: 0.603591 Accuracy: 87.84% 5 Validation loss: 12683.649414 Best loss: 0.603591 Accuracy: 78.30% 6 Validation loss: 1271.122559 Best loss: 0.603591 Accuracy: 92.96% 7 Validation loss: 2100.986572 Best loss: 0.603591 Accuracy: 85.89% 8 Validation loss: 828.236633 Best loss: 0.603591 Accuracy: 94.92% 9 Validation loss: 2356.615723 Best loss: 0.603591 Accuracy: 90.73% 10 Validation loss: 822.293030 Best loss: 0.603591 Accuracy: 91.95% 11 Validation loss: 297814.593750 Best loss: 0.603591 Accuracy: 88.04% 12 Validation loss: 212198.906250 Best loss: 0.603591 Accuracy: 93.20% 13 Validation loss: 95973.187500 Best loss: 0.603591 Accuracy: 90.97% 14 Validation loss: 83211.187500 Best loss: 0.603591 Accuracy: 94.45% 15 Validation loss: 31138.560547 Best loss: 0.603591 Accuracy: 94.84% 16 Validation loss: 41290.054688 Best loss: 0.603591 Accuracy: 95.66% 17 Validation loss: 21150.535156 Best loss: 0.603591 Accuracy: 95.86% 18 Validation loss: 25854.531250 Best loss: 0.603591 Accuracy: 96.09% 19 Validation loss: 12728.083984 Best loss: 0.603591 Accuracy: 95.00% 20 Validation loss: 14436.103516 Best loss: 0.603591 Accuracy: 94.45% 21 Validation loss: 9011.983398 Best loss: 0.603591 Accuracy: 95.11% 22 Validation loss: 2826479.750000 Best loss: 0.603591 Accuracy: 92.22% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 29.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 0.613072 Best loss: 0.613072 Accuracy: 93.67% 1 Validation loss: 0.259798 Best loss: 0.259798 Accuracy: 94.29% 2 Validation loss: 0.281153 Best loss: 0.259798 Accuracy: 95.00% 3 Validation loss: 0.268049 Best loss: 0.259798 Accuracy: 95.19% 4 Validation loss: 0.182030 Best loss: 0.182030 Accuracy: 95.27% 5 Validation loss: 0.186816 Best loss: 0.182030 Accuracy: 95.39% 6 Validation loss: 213030.031250 Best loss: 0.182030 Accuracy: 70.25% 7 Validation loss: 29767.236328 Best loss: 0.182030 Accuracy: 81.98% 8 Validation loss: 9373.758789 Best loss: 0.182030 Accuracy: 86.40% 9 Validation loss: 17284.351562 Best loss: 0.182030 Accuracy: 74.94% 10 Validation loss: 14326.447266 Best loss: 0.182030 Accuracy: 82.49% 11 Validation loss: 7941.537109 Best loss: 0.182030 Accuracy: 89.76% 12 Validation loss: 11600.221680 Best loss: 0.182030 Accuracy: 83.23% 13 Validation loss: 7810.464355 Best loss: 0.182030 Accuracy: 90.89% 14 Validation loss: 4595.671875 Best loss: 0.182030 Accuracy: 88.98% 15 Validation loss: 15760.617188 Best loss: 0.182030 Accuracy: 85.46% 16 Validation loss: 1081191.250000 Best loss: 0.182030 Accuracy: 81.63% 17 Validation loss: 799773.062500 Best loss: 0.182030 Accuracy: 67.71% 18 Validation loss: 400206.281250 Best loss: 0.182030 Accuracy: 87.22% 19 Validation loss: 67457.335938 Best loss: 0.182030 Accuracy: 93.90% 20 Validation loss: 164652.062500 Best loss: 0.182030 Accuracy: 92.26% 21 Validation loss: 124119.445312 Best loss: 0.182030 Accuracy: 83.74% 22 Validation loss: 45853.121094 Best loss: 0.182030 Accuracy: 94.41% 23 Validation loss: 7511509.000000 Best loss: 0.182030 Accuracy: 89.60% 24 Validation loss: 499797.781250 Best loss: 0.182030 Accuracy: 91.24% 25 Validation loss: 402497.406250 Best loss: 0.182030 Accuracy: 88.66% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 33.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 1.950097 Best loss: 1.950097 Accuracy: 59.70% 1 Validation loss: 139282.093750 Best loss: 1.950097 Accuracy: 45.82% 2 Validation loss: 8358.582031 Best loss: 1.950097 Accuracy: 71.23% 3 Validation loss: 1902.793213 Best loss: 1.950097 Accuracy: 74.55% 4 Validation loss: 792.501160 Best loss: 1.950097 Accuracy: 82.33% 5 Validation loss: 458.181549 Best loss: 1.950097 Accuracy: 91.48% 6 Validation loss: 1508.993408 Best loss: 1.950097 Accuracy: 78.15% 7 Validation loss: 4473517.500000 Best loss: 1.950097 Accuracy: 69.98% 8 Validation loss: 860445.000000 Best loss: 1.950097 Accuracy: 75.45% 9 Validation loss: 808339.562500 Best loss: 1.950097 Accuracy: 88.19% 10 Validation loss: 174375.562500 Best loss: 1.950097 Accuracy: 88.43% 11 Validation loss: 143580.515625 Best loss: 1.950097 Accuracy: 90.34% 12 Validation loss: 516631.062500 Best loss: 1.950097 Accuracy: 74.82% 13 Validation loss: 100200.875000 Best loss: 1.950097 Accuracy: 87.37% 14 Validation loss: 73592.945312 Best loss: 1.950097 Accuracy: 83.50% 15 Validation loss: 32361.263672 Best loss: 1.950097 Accuracy: 91.09% 16 Validation loss: 76126.289062 Best loss: 1.950097 Accuracy: 90.38% 17 Validation loss: 67864.156250 Best loss: 1.950097 Accuracy: 93.00% 18 Validation loss: 56283.777344 Best loss: 1.950097 Accuracy: 90.27% 19 Validation loss: 60469.761719 Best loss: 1.950097 Accuracy: 93.51% 20 Validation loss: 26907.525391 Best loss: 1.950097 Accuracy: 93.24% 21 Validation loss: 88597.882812 Best loss: 1.950097 Accuracy: 88.90% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 28.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.135344 Best loss: 0.135344 Accuracy: 96.33% 1 Validation loss: 0.147649 Best loss: 0.135344 Accuracy: 96.01% 2 Validation loss: 0.217860 Best loss: 0.135344 Accuracy: 94.41% 3 Validation loss: 0.217625 Best loss: 0.135344 Accuracy: 92.22% 4 Validation loss: 0.353568 Best loss: 0.135344 Accuracy: 87.57% 5 Validation loss: 0.393556 Best loss: 0.135344 Accuracy: 91.52% 6 Validation loss: 1.043746 Best loss: 0.135344 Accuracy: 73.22% 7 Validation loss: 0.389935 Best loss: 0.135344 Accuracy: 88.47% 8 Validation loss: 0.241502 Best loss: 0.135344 Accuracy: 91.40% 9 Validation loss: 0.242850 Best loss: 0.135344 Accuracy: 92.96% 10 Validation loss: 0.188783 Best loss: 0.135344 Accuracy: 95.62% 11 Validation loss: 0.202283 Best loss: 0.135344 Accuracy: 94.18% 12 Validation loss: 0.193989 Best loss: 0.135344 Accuracy: 95.00% 13 Validation loss: 0.235340 Best loss: 0.135344 Accuracy: 94.68% 14 Validation loss: 0.210484 Best loss: 0.135344 Accuracy: 93.20% 15 Validation loss: 2.901139 Best loss: 0.135344 Accuracy: 87.80% 16 Validation loss: 38.109409 Best loss: 0.135344 Accuracy: 53.48% 17 Validation loss: 2.266012 Best loss: 0.135344 Accuracy: 71.34% 18 Validation loss: 1.439322 Best loss: 0.135344 Accuracy: 69.82% 19 Validation loss: 1.708308 Best loss: 0.135344 Accuracy: 63.33% 20 Validation loss: 2.087500 Best loss: 0.135344 Accuracy: 64.62% 21 Validation loss: 0.874223 Best loss: 0.135344 Accuracy: 79.20% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 15.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.125185 Best loss: 0.125185 Accuracy: 96.44% 1 Validation loss: 0.190503 Best loss: 0.125185 Accuracy: 95.27% 2 Validation loss: 0.357868 Best loss: 0.125185 Accuracy: 87.76% 3 Validation loss: 0.475410 Best loss: 0.125185 Accuracy: 82.21% 4 Validation loss: 0.328553 Best loss: 0.125185 Accuracy: 93.16% 5 Validation loss: 0.332784 Best loss: 0.125185 Accuracy: 85.61% 6 Validation loss: 0.218854 Best loss: 0.125185 Accuracy: 92.92% 7 Validation loss: 0.300952 Best loss: 0.125185 Accuracy: 92.42% 8 Validation loss: 3.071088 Best loss: 0.125185 Accuracy: 88.15% 9 Validation loss: 0.616220 Best loss: 0.125185 Accuracy: 76.70% 10 Validation loss: 0.803335 Best loss: 0.125185 Accuracy: 71.46% 11 Validation loss: 1.036310 Best loss: 0.125185 Accuracy: 73.96% 12 Validation loss: 0.595329 Best loss: 0.125185 Accuracy: 74.08% 13 Validation loss: 0.613172 Best loss: 0.125185 Accuracy: 76.62% 14 Validation loss: 0.491878 Best loss: 0.125185 Accuracy: 77.44% 15 Validation loss: 0.468614 Best loss: 0.125185 Accuracy: 78.77% 16 Validation loss: 0.548663 Best loss: 0.125185 Accuracy: 74.08% 17 Validation loss: 0.517325 Best loss: 0.125185 Accuracy: 76.74% 18 Validation loss: 0.777264 Best loss: 0.125185 Accuracy: 75.33% 19 Validation loss: 1.434856 Best loss: 0.125185 Accuracy: 75.76% 20 Validation loss: 0.511783 Best loss: 0.125185 Accuracy: 80.34% 21 Validation loss: 0.585553 Best loss: 0.125185 Accuracy: 77.05% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 15.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.141416 Best loss: 0.141416 Accuracy: 96.72% 1 Validation loss: 0.129403 Best loss: 0.129403 Accuracy: 96.76% 2 Validation loss: 0.573770 Best loss: 0.129403 Accuracy: 72.60% 3 Validation loss: 0.252816 Best loss: 0.129403 Accuracy: 93.75% 4 Validation loss: 0.272212 Best loss: 0.129403 Accuracy: 92.49% 5 Validation loss: 2.345474 Best loss: 0.129403 Accuracy: 86.86% 6 Validation loss: 1.936732 Best loss: 0.129403 Accuracy: 49.30% 7 Validation loss: 2.590318 Best loss: 0.129403 Accuracy: 48.28% 8 Validation loss: 0.867677 Best loss: 0.129403 Accuracy: 71.81% 9 Validation loss: 1.017916 Best loss: 0.129403 Accuracy: 54.38% 10 Validation loss: 0.689297 Best loss: 0.129403 Accuracy: 73.81% 11 Validation loss: 0.638448 Best loss: 0.129403 Accuracy: 79.05% 12 Validation loss: 0.636605 Best loss: 0.129403 Accuracy: 69.23% 13 Validation loss: 0.603535 Best loss: 0.129403 Accuracy: 79.28% 14 Validation loss: 23.723230 Best loss: 0.129403 Accuracy: 26.27% 15 Validation loss: 1.687600 Best loss: 0.129403 Accuracy: 43.24% 16 Validation loss: 1.361266 Best loss: 0.129403 Accuracy: 50.74% 17 Validation loss: 1.002386 Best loss: 0.129403 Accuracy: 53.05% 18 Validation loss: 1.021766 Best loss: 0.129403 Accuracy: 50.82% 19 Validation loss: 1.061884 Best loss: 0.129403 Accuracy: 54.42% 20 Validation loss: 0.984973 Best loss: 0.129403 Accuracy: 56.18% 21 Validation loss: 0.953650 Best loss: 0.129403 Accuracy: 56.76% 22 Validation loss: 1.042044 Best loss: 0.129403 Accuracy: 55.71% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=50, batch_size=100, learning_rate=0.02, dropout_rate=0.3, total= 16.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.212757 Best loss: 0.212757 Accuracy: 94.72% 1 Validation loss: 0.191348 Best loss: 0.191348 Accuracy: 94.80% 2 Validation loss: 0.198261 Best loss: 0.191348 Accuracy: 95.15% 3 Validation loss: 0.199836 Best loss: 0.191348 Accuracy: 95.11% 4 Validation loss: 0.212903 Best loss: 0.191348 Accuracy: 94.14% 5 Validation loss: 0.204123 Best loss: 0.191348 Accuracy: 93.12% 6 Validation loss: 0.174200 Best loss: 0.174200 Accuracy: 94.10% 7 Validation loss: 0.207012 Best loss: 0.174200 Accuracy: 93.55% 8 Validation loss: 0.302494 Best loss: 0.174200 Accuracy: 87.57% 9 Validation loss: 0.209350 Best loss: 0.174200 Accuracy: 92.73% 10 Validation loss: 0.455979 Best loss: 0.174200 Accuracy: 78.54% 11 Validation loss: 0.332775 Best loss: 0.174200 Accuracy: 90.38% 12 Validation loss: 0.273478 Best loss: 0.174200 Accuracy: 91.91% 13 Validation loss: 0.269668 Best loss: 0.174200 Accuracy: 90.30% 14 Validation loss: 0.316981 Best loss: 0.174200 Accuracy: 88.35% 15 Validation loss: 0.399193 Best loss: 0.174200 Accuracy: 84.17% 16 Validation loss: 0.369989 Best loss: 0.174200 Accuracy: 88.55% 17 Validation loss: 0.312551 Best loss: 0.174200 Accuracy: 90.58% 18 Validation loss: 0.368975 Best loss: 0.174200 Accuracy: 86.71% 19 Validation loss: 0.449709 Best loss: 0.174200 Accuracy: 90.54% 20 Validation loss: 0.519887 Best loss: 0.174200 Accuracy: 90.73% 21 Validation loss: 0.356092 Best loss: 0.174200 Accuracy: 94.72% 22 Validation loss: 0.370779 Best loss: 0.174200 Accuracy: 94.96% 23 Validation loss: 0.456513 Best loss: 0.174200 Accuracy: 89.13% 24 Validation loss: 0.590186 Best loss: 0.174200 Accuracy: 78.85% 25 Validation loss: 0.784956 Best loss: 0.174200 Accuracy: 62.74% 26 Validation loss: 0.662596 Best loss: 0.174200 Accuracy: 81.08% 27 Validation loss: 0.636694 Best loss: 0.174200 Accuracy: 78.03% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 19.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.302077 Best loss: 0.302077 Accuracy: 92.92% 1 Validation loss: 0.210691 Best loss: 0.210691 Accuracy: 94.49% 2 Validation loss: 0.193434 Best loss: 0.193434 Accuracy: 94.72% 3 Validation loss: 0.184382 Best loss: 0.184382 Accuracy: 94.18% 4 Validation loss: 0.213611 Best loss: 0.184382 Accuracy: 93.63% 5 Validation loss: 0.184989 Best loss: 0.184382 Accuracy: 94.64% 6 Validation loss: 0.204263 Best loss: 0.184382 Accuracy: 92.53% 7 Validation loss: 0.217445 Best loss: 0.184382 Accuracy: 93.12% 8 Validation loss: 0.223440 Best loss: 0.184382 Accuracy: 94.18% 9 Validation loss: 0.286126 Best loss: 0.184382 Accuracy: 90.42% 10 Validation loss: 0.246122 Best loss: 0.184382 Accuracy: 93.00% 11 Validation loss: 0.259003 Best loss: 0.184382 Accuracy: 90.77% 12 Validation loss: 0.288321 Best loss: 0.184382 Accuracy: 89.13% 13 Validation loss: 0.276695 Best loss: 0.184382 Accuracy: 89.68% 14 Validation loss: 0.262103 Best loss: 0.184382 Accuracy: 90.70% 15 Validation loss: 0.277833 Best loss: 0.184382 Accuracy: 89.64% 16 Validation loss: 0.213854 Best loss: 0.184382 Accuracy: 94.68% 17 Validation loss: 0.252267 Best loss: 0.184382 Accuracy: 92.46% 18 Validation loss: 0.316104 Best loss: 0.184382 Accuracy: 87.72% 19 Validation loss: 0.486578 Best loss: 0.184382 Accuracy: 80.34% 20 Validation loss: 0.432241 Best loss: 0.184382 Accuracy: 91.56% 21 Validation loss: 0.323893 Best loss: 0.184382 Accuracy: 90.85% 22 Validation loss: 0.393760 Best loss: 0.184382 Accuracy: 88.39% 23 Validation loss: 0.296299 Best loss: 0.184382 Accuracy: 89.13% 24 Validation loss: 0.384863 Best loss: 0.184382 Accuracy: 89.41% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 17.3s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.199610 Best loss: 0.199610 Accuracy: 94.45% 1 Validation loss: 0.332410 Best loss: 0.199610 Accuracy: 94.45% 2 Validation loss: 0.191885 Best loss: 0.191885 Accuracy: 94.49% 3 Validation loss: 0.177653 Best loss: 0.177653 Accuracy: 95.66% 4 Validation loss: 0.207591 Best loss: 0.177653 Accuracy: 94.02% 5 Validation loss: 0.187438 Best loss: 0.177653 Accuracy: 95.19% 6 Validation loss: 0.221946 Best loss: 0.177653 Accuracy: 92.49% 7 Validation loss: 0.261466 Best loss: 0.177653 Accuracy: 89.91% 8 Validation loss: 0.245463 Best loss: 0.177653 Accuracy: 92.77% 9 Validation loss: 0.235556 Best loss: 0.177653 Accuracy: 94.49% 10 Validation loss: 0.226626 Best loss: 0.177653 Accuracy: 92.73% 11 Validation loss: 0.242238 Best loss: 0.177653 Accuracy: 91.79% 12 Validation loss: 0.209356 Best loss: 0.177653 Accuracy: 93.35% 13 Validation loss: 0.327541 Best loss: 0.177653 Accuracy: 89.44% 14 Validation loss: 0.268664 Best loss: 0.177653 Accuracy: 92.77% 15 Validation loss: 0.278430 Best loss: 0.177653 Accuracy: 88.43% 16 Validation loss: 0.330117 Best loss: 0.177653 Accuracy: 91.48% 17 Validation loss: 0.330838 Best loss: 0.177653 Accuracy: 88.47% 18 Validation loss: 0.273336 Best loss: 0.177653 Accuracy: 90.93% 19 Validation loss: 0.406527 Best loss: 0.177653 Accuracy: 84.01% 20 Validation loss: 0.295914 Best loss: 0.177653 Accuracy: 92.61% 21 Validation loss: 0.362588 Best loss: 0.177653 Accuracy: 85.42% 22 Validation loss: 0.286953 Best loss: 0.177653 Accuracy: 89.13% 23 Validation loss: 0.231235 Best loss: 0.177653 Accuracy: 93.04% 24 Validation loss: 0.405763 Best loss: 0.177653 Accuracy: 87.10% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=160, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 17.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.745659 Best loss: 1.745659 Accuracy: 18.73% 1 Validation loss: 1.635658 Best loss: 1.635658 Accuracy: 20.91% 2 Validation loss: 1.754412 Best loss: 1.635658 Accuracy: 18.73% 3 Validation loss: 1.641608 Best loss: 1.635658 Accuracy: 22.01% 4 Validation loss: 1.811811 Best loss: 1.635658 Accuracy: 19.27% 5 Validation loss: 1.727053 Best loss: 1.635658 Accuracy: 19.08% 6 Validation loss: 1.924918 Best loss: 1.635658 Accuracy: 19.27% 7 Validation loss: 1.825212 Best loss: 1.635658 Accuracy: 19.08% 8 Validation loss: 1.931508 Best loss: 1.635658 Accuracy: 22.01% 9 Validation loss: 1.693486 Best loss: 1.635658 Accuracy: 22.01% 10 Validation loss: 1.948973 Best loss: 1.635658 Accuracy: 19.08% 11 Validation loss: 2.203250 Best loss: 1.635658 Accuracy: 18.73% 12 Validation loss: 3.654564 Best loss: 1.635658 Accuracy: 19.27% 13 Validation loss: 1.773366 Best loss: 1.635658 Accuracy: 22.01% 14 Validation loss: 1.764535 Best loss: 1.635658 Accuracy: 19.08% 15 Validation loss: 1.653688 Best loss: 1.635658 Accuracy: 22.01% 16 Validation loss: 1.795667 Best loss: 1.635658 Accuracy: 19.08% 17 Validation loss: 1.620414 Best loss: 1.620414 Accuracy: 20.91% 18 Validation loss: 1.857953 Best loss: 1.620414 Accuracy: 19.08% 19 Validation loss: 1.799152 Best loss: 1.620414 Accuracy: 22.01% 20 Validation loss: 1.714771 Best loss: 1.620414 Accuracy: 18.73% 21 Validation loss: 1.773471 Best loss: 1.620414 Accuracy: 20.91% 22 Validation loss: 1.659323 Best loss: 1.620414 Accuracy: 18.73% 23 Validation loss: 1.735742 Best loss: 1.620414 Accuracy: 20.91% 24 Validation loss: 1.916227 Best loss: 1.620414 Accuracy: 19.08% 25 Validation loss: 1.673684 Best loss: 1.620414 Accuracy: 20.91% 26 Validation loss: 1.692085 Best loss: 1.620414 Accuracy: 19.08% 27 Validation loss: 2.016028 Best loss: 1.620414 Accuracy: 18.73% 28 Validation loss: 2.093408 Best loss: 1.620414 Accuracy: 18.73% 29 Validation loss: 1.780830 Best loss: 1.620414 Accuracy: 20.91% 30 Validation loss: 1.943406 Best loss: 1.620414 Accuracy: 22.01% 31 Validation loss: 1.728307 Best loss: 1.620414 Accuracy: 19.27% 32 Validation loss: 1.743058 Best loss: 1.620414 Accuracy: 22.01% 33 Validation loss: 1.762696 Best loss: 1.620414 Accuracy: 22.01% 34 Validation loss: 1.836625 Best loss: 1.620414 Accuracy: 19.08% 35 Validation loss: 2.363962 Best loss: 1.620414 Accuracy: 19.27% 36 Validation loss: 2.000119 Best loss: 1.620414 Accuracy: 22.01% 37 Validation loss: 1.841915 Best loss: 1.620414 Accuracy: 19.08% 38 Validation loss: 1.780878 Best loss: 1.620414 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 3.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.685628 Best loss: 1.685628 Accuracy: 19.27% 1 Validation loss: 1.804651 Best loss: 1.685628 Accuracy: 22.01% 2 Validation loss: 1.968936 Best loss: 1.685628 Accuracy: 18.73% 3 Validation loss: 1.840774 Best loss: 1.685628 Accuracy: 22.01% 4 Validation loss: 1.861938 Best loss: 1.685628 Accuracy: 19.08% 5 Validation loss: 1.689674 Best loss: 1.685628 Accuracy: 18.73% 6 Validation loss: 1.906064 Best loss: 1.685628 Accuracy: 20.91% 7 Validation loss: 1.984243 Best loss: 1.685628 Accuracy: 19.08% 8 Validation loss: 1.813587 Best loss: 1.685628 Accuracy: 19.08% 9 Validation loss: 1.727997 Best loss: 1.685628 Accuracy: 20.91% 10 Validation loss: 1.857154 Best loss: 1.685628 Accuracy: 20.91% 11 Validation loss: 1.863634 Best loss: 1.685628 Accuracy: 19.08% 12 Validation loss: 1.961647 Best loss: 1.685628 Accuracy: 20.91% 13 Validation loss: 2.264751 Best loss: 1.685628 Accuracy: 22.01% 14 Validation loss: 1.889982 Best loss: 1.685628 Accuracy: 19.27% 15 Validation loss: 1.966228 Best loss: 1.685628 Accuracy: 20.91% 16 Validation loss: 1.819791 Best loss: 1.685628 Accuracy: 19.08% 17 Validation loss: 2.039490 Best loss: 1.685628 Accuracy: 18.73% 18 Validation loss: 1.844752 Best loss: 1.685628 Accuracy: 19.08% 19 Validation loss: 1.887733 Best loss: 1.685628 Accuracy: 19.27% 20 Validation loss: 2.087358 Best loss: 1.685628 Accuracy: 19.27% 21 Validation loss: 1.771424 Best loss: 1.685628 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 1.9min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1.855662 Best loss: 1.855662 Accuracy: 22.01% 1 Validation loss: 2.010948 Best loss: 1.855662 Accuracy: 19.27% 2 Validation loss: 1.961876 Best loss: 1.855662 Accuracy: 19.08% 3 Validation loss: 1.831088 Best loss: 1.831088 Accuracy: 19.27% 4 Validation loss: 2.706486 Best loss: 1.831088 Accuracy: 19.08% 5 Validation loss: 1.794573 Best loss: 1.794573 Accuracy: 20.91% 6 Validation loss: 1.869709 Best loss: 1.794573 Accuracy: 19.27% 7 Validation loss: 2.283661 Best loss: 1.794573 Accuracy: 18.73% 8 Validation loss: 1.899324 Best loss: 1.794573 Accuracy: 19.27% 9 Validation loss: 2.216293 Best loss: 1.794573 Accuracy: 19.27% 10 Validation loss: 1.937226 Best loss: 1.794573 Accuracy: 18.73% 11 Validation loss: 2.055965 Best loss: 1.794573 Accuracy: 19.27% 12 Validation loss: 2.173638 Best loss: 1.794573 Accuracy: 20.91% 13 Validation loss: 3.022261 Best loss: 1.794573 Accuracy: 19.08% 14 Validation loss: 1.950351 Best loss: 1.794573 Accuracy: 22.01% 15 Validation loss: 1.688134 Best loss: 1.688134 Accuracy: 19.27% 16 Validation loss: 2.224023 Best loss: 1.688134 Accuracy: 18.73% 17 Validation loss: 2.050119 Best loss: 1.688134 Accuracy: 22.01% 18 Validation loss: 2.232249 Best loss: 1.688134 Accuracy: 19.08% 19 Validation loss: 2.014817 Best loss: 1.688134 Accuracy: 18.73% 20 Validation loss: 1.814239 Best loss: 1.688134 Accuracy: 19.27% 21 Validation loss: 2.084231 Best loss: 1.688134 Accuracy: 22.01% 22 Validation loss: 1.870279 Best loss: 1.688134 Accuracy: 20.91% 23 Validation loss: 2.219250 Best loss: 1.688134 Accuracy: 22.01% 24 Validation loss: 1.701719 Best loss: 1.688134 Accuracy: 18.73% 25 Validation loss: 1.822189 Best loss: 1.688134 Accuracy: 22.01% 26 Validation loss: 1.659310 Best loss: 1.659310 Accuracy: 18.73% 27 Validation loss: 1.696233 Best loss: 1.659310 Accuracy: 18.73% 28 Validation loss: 2.340528 Best loss: 1.659310 Accuracy: 19.08% 29 Validation loss: 2.556751 Best loss: 1.659310 Accuracy: 20.91% 30 Validation loss: 2.089121 Best loss: 1.659310 Accuracy: 19.27% 31 Validation loss: 1.874208 Best loss: 1.659310 Accuracy: 22.01% 32 Validation loss: 1.977130 Best loss: 1.659310 Accuracy: 19.08% 33 Validation loss: 2.284336 Best loss: 1.659310 Accuracy: 22.01% 34 Validation loss: 2.327761 Best loss: 1.659310 Accuracy: 19.27% 35 Validation loss: 1.680173 Best loss: 1.659310 Accuracy: 19.08% 36 Validation loss: 1.822019 Best loss: 1.659310 Accuracy: 20.91% 37 Validation loss: 2.149864 Best loss: 1.659310 Accuracy: 19.08% 38 Validation loss: 2.686031 Best loss: 1.659310 Accuracy: 18.73% 39 Validation loss: 2.304118 Best loss: 1.659310 Accuracy: 19.27% 40 Validation loss: 1.856371 Best loss: 1.659310 Accuracy: 19.08% 41 Validation loss: 2.028929 Best loss: 1.659310 Accuracy: 22.01% 42 Validation loss: 2.733980 Best loss: 1.659310 Accuracy: 22.01% 43 Validation loss: 2.957522 Best loss: 1.659310 Accuracy: 20.91% 44 Validation loss: 1.912538 Best loss: 1.659310 Accuracy: 18.73% 45 Validation loss: 1.725067 Best loss: 1.659310 Accuracy: 19.27% 46 Validation loss: 2.622473 Best loss: 1.659310 Accuracy: 20.91% 47 Validation loss: 2.118242 Best loss: 1.659310 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=50, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 4.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 952025.562500 Best loss: 952025.562500 Accuracy: 21.07% 1 Validation loss: 1254392.125000 Best loss: 952025.562500 Accuracy: 18.73% 2 Validation loss: 368714.718750 Best loss: 368714.718750 Accuracy: 17.90% 3 Validation loss: 16503836.000000 Best loss: 368714.718750 Accuracy: 19.08% 4 Validation loss: 110613600.000000 Best loss: 368714.718750 Accuracy: 19.08% 5 Validation loss: 9089108.000000 Best loss: 368714.718750 Accuracy: 18.73% 6 Validation loss: 3078665.250000 Best loss: 368714.718750 Accuracy: 18.73% 7 Validation loss: 10775599.000000 Best loss: 368714.718750 Accuracy: 22.01% 8 Validation loss: 9269257.000000 Best loss: 368714.718750 Accuracy: 18.73% 9 Validation loss: 7435135.000000 Best loss: 368714.718750 Accuracy: 19.27% 10 Validation loss: 14555714.000000 Best loss: 368714.718750 Accuracy: 19.27% 11 Validation loss: 3213492.000000 Best loss: 368714.718750 Accuracy: 20.91% 12 Validation loss: 47409196.000000 Best loss: 368714.718750 Accuracy: 20.91% 13 Validation loss: 5604255.000000 Best loss: 368714.718750 Accuracy: 32.21% 14 Validation loss: 2850620.750000 Best loss: 368714.718750 Accuracy: 19.08% 15 Validation loss: 16504910.000000 Best loss: 368714.718750 Accuracy: 19.08% 16 Validation loss: 31930530.000000 Best loss: 368714.718750 Accuracy: 19.08% 17 Validation loss: 90792552.000000 Best loss: 368714.718750 Accuracy: 19.27% 18 Validation loss: 19310058.000000 Best loss: 368714.718750 Accuracy: 18.73% 19 Validation loss: 5648837.000000 Best loss: 368714.718750 Accuracy: 22.01% 20 Validation loss: 2640695.750000 Best loss: 368714.718750 Accuracy: 20.91% 21 Validation loss: 5278120.500000 Best loss: 368714.718750 Accuracy: 18.73% 22 Validation loss: 7584829.500000 Best loss: 368714.718750 Accuracy: 19.12% 23 Validation loss: 47340232.000000 Best loss: 368714.718750 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 2.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 391715.187500 Best loss: 391715.187500 Accuracy: 19.08% 1 Validation loss: 46187164.000000 Best loss: 391715.187500 Accuracy: 19.08% 2 Validation loss: 5020659.500000 Best loss: 391715.187500 Accuracy: 20.91% 3 Validation loss: 28915198.000000 Best loss: 391715.187500 Accuracy: 19.08% 4 Validation loss: 5910688.000000 Best loss: 391715.187500 Accuracy: 19.27% 5 Validation loss: 3631825.750000 Best loss: 391715.187500 Accuracy: 19.47% 6 Validation loss: 1128271.125000 Best loss: 391715.187500 Accuracy: 19.08% 7 Validation loss: 8031980.000000 Best loss: 391715.187500 Accuracy: 18.73% 8 Validation loss: 990393.062500 Best loss: 391715.187500 Accuracy: 19.08% 9 Validation loss: 5453741.500000 Best loss: 391715.187500 Accuracy: 19.27% 10 Validation loss: 1568217.000000 Best loss: 391715.187500 Accuracy: 14.93% 11 Validation loss: 46709308.000000 Best loss: 391715.187500 Accuracy: 22.01% 12 Validation loss: 28269980.000000 Best loss: 391715.187500 Accuracy: 20.91% 13 Validation loss: 11795434.000000 Best loss: 391715.187500 Accuracy: 19.08% 14 Validation loss: 6142531.500000 Best loss: 391715.187500 Accuracy: 19.08% 15 Validation loss: 13588219.000000 Best loss: 391715.187500 Accuracy: 18.73% 16 Validation loss: 4374987.000000 Best loss: 391715.187500 Accuracy: 19.27% 17 Validation loss: 45533912.000000 Best loss: 391715.187500 Accuracy: 19.08% 18 Validation loss: 5358625.000000 Best loss: 391715.187500 Accuracy: 19.08% 19 Validation loss: 48387468.000000 Best loss: 391715.187500 Accuracy: 18.73% 20 Validation loss: 2984286.250000 Best loss: 391715.187500 Accuracy: 22.01% 21 Validation loss: 21895612.000000 Best loss: 391715.187500 Accuracy: 22.01% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5 0 Validation loss: 1118213.125000 Best loss: 1118213.125000 Accuracy: 18.73% 1 Validation loss: 5055770.000000 Best loss: 1118213.125000 Accuracy: 18.73% 2 Validation loss: 14245137.000000 Best loss: 1118213.125000 Accuracy: 18.73% 3 Validation loss: 2433383.000000 Best loss: 1118213.125000 Accuracy: 18.84% 4 Validation loss: 61496620.000000 Best loss: 1118213.125000 Accuracy: 19.08% 5 Validation loss: 9929818.000000 Best loss: 1118213.125000 Accuracy: 20.91% 6 Validation loss: 10575771.000000 Best loss: 1118213.125000 Accuracy: 19.12% 7 Validation loss: 3574038.500000 Best loss: 1118213.125000 Accuracy: 19.27% 8 Validation loss: 3214543.750000 Best loss: 1118213.125000 Accuracy: 19.27% 9 Validation loss: 7764833.500000 Best loss: 1118213.125000 Accuracy: 22.01% 10 Validation loss: 4330503.000000 Best loss: 1118213.125000 Accuracy: 18.73% 11 Validation loss: 3070537.750000 Best loss: 1118213.125000 Accuracy: 18.73% 12 Validation loss: 3275350.750000 Best loss: 1118213.125000 Accuracy: 20.91% 13 Validation loss: 24808400.000000 Best loss: 1118213.125000 Accuracy: 22.01% 14 Validation loss: 1655936.750000 Best loss: 1118213.125000 Accuracy: 18.73% 15 Validation loss: 19162144.000000 Best loss: 1118213.125000 Accuracy: 20.91% 16 Validation loss: 4411918.000000 Best loss: 1118213.125000 Accuracy: 18.73% 17 Validation loss: 2718380.500000 Best loss: 1118213.125000 Accuracy: 18.73% 18 Validation loss: 44280272.000000 Best loss: 1118213.125000 Accuracy: 19.08% 19 Validation loss: 85360448.000000 Best loss: 1118213.125000 Accuracy: 20.91% 20 Validation loss: 5739983.500000 Best loss: 1118213.125000 Accuracy: 18.73% 21 Validation loss: 2587777.750000 Best loss: 1118213.125000 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=90, batch_size=10, learning_rate=0.1, dropout_rate=0.5, total= 2.2min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.555885 Best loss: 1.555885 Accuracy: 22.75% 1 Validation loss: 1.613777 Best loss: 1.555885 Accuracy: 19.27% 2 Validation loss: 1.610008 Best loss: 1.555885 Accuracy: 22.01% 3 Validation loss: 1.607947 Best loss: 1.555885 Accuracy: 22.01% 4 Validation loss: 1.619505 Best loss: 1.555885 Accuracy: 22.01% 5 Validation loss: 1.616166 Best loss: 1.555885 Accuracy: 22.01% 6 Validation loss: 1.613033 Best loss: 1.555885 Accuracy: 19.27% 7 Validation loss: 1.615293 Best loss: 1.555885 Accuracy: 22.01% 8 Validation loss: 1.621618 Best loss: 1.555885 Accuracy: 19.27% 9 Validation loss: 1.611582 Best loss: 1.555885 Accuracy: 22.01% 10 Validation loss: 1.608235 Best loss: 1.555885 Accuracy: 22.01% 11 Validation loss: 1.609513 Best loss: 1.555885 Accuracy: 22.01% 12 Validation loss: 1.608571 Best loss: 1.555885 Accuracy: 22.01% 13 Validation loss: 1.612131 Best loss: 1.555885 Accuracy: 18.73% 14 Validation loss: 1.610412 Best loss: 1.555885 Accuracy: 22.01% 15 Validation loss: 1.609290 Best loss: 1.555885 Accuracy: 22.01% 16 Validation loss: 1.608220 Best loss: 1.555885 Accuracy: 22.01% 17 Validation loss: 1.611068 Best loss: 1.555885 Accuracy: 19.27% 18 Validation loss: 1.614791 Best loss: 1.555885 Accuracy: 22.01% 19 Validation loss: 1.610238 Best loss: 1.555885 Accuracy: 19.08% 20 Validation loss: 1.610642 Best loss: 1.555885 Accuracy: 20.91% 21 Validation loss: 1.617286 Best loss: 1.555885 Accuracy: 19.27% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 2.0min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.613244 Best loss: 1.613244 Accuracy: 22.01% 1 Validation loss: 1.615557 Best loss: 1.613244 Accuracy: 19.27% 2 Validation loss: 1.609281 Best loss: 1.609281 Accuracy: 22.01% 3 Validation loss: 1.613823 Best loss: 1.609281 Accuracy: 18.73% 4 Validation loss: 1.612018 Best loss: 1.609281 Accuracy: 22.01% 5 Validation loss: 1.610147 Best loss: 1.609281 Accuracy: 22.01% 6 Validation loss: 1.621746 Best loss: 1.609281 Accuracy: 19.08% 7 Validation loss: 1.612068 Best loss: 1.609281 Accuracy: 18.73% 8 Validation loss: 1.611325 Best loss: 1.609281 Accuracy: 22.01% 9 Validation loss: 1.608815 Best loss: 1.608815 Accuracy: 22.01% 10 Validation loss: 1.610081 Best loss: 1.608815 Accuracy: 22.01% 11 Validation loss: 1.610640 Best loss: 1.608815 Accuracy: 22.01% 12 Validation loss: 1.610745 Best loss: 1.608815 Accuracy: 19.27% 13 Validation loss: 1.611638 Best loss: 1.608815 Accuracy: 19.08% 14 Validation loss: 1.608832 Best loss: 1.608815 Accuracy: 22.01% 15 Validation loss: 1.608106 Best loss: 1.608106 Accuracy: 22.01% 16 Validation loss: 1.613543 Best loss: 1.608106 Accuracy: 22.01% 17 Validation loss: 1.611918 Best loss: 1.608106 Accuracy: 18.73% 18 Validation loss: 1.614609 Best loss: 1.608106 Accuracy: 22.01% 19 Validation loss: 1.608659 Best loss: 1.608106 Accuracy: 22.01% 20 Validation loss: 1.609336 Best loss: 1.608106 Accuracy: 22.01% 21 Validation loss: 1.619996 Best loss: 1.608106 Accuracy: 22.01% 22 Validation loss: 1.620808 Best loss: 1.608106 Accuracy: 19.27% 23 Validation loss: 1.618630 Best loss: 1.608106 Accuracy: 22.01% 24 Validation loss: 1.613378 Best loss: 1.608106 Accuracy: 22.01% 25 Validation loss: 1.608673 Best loss: 1.608106 Accuracy: 22.01% 26 Validation loss: 1.608852 Best loss: 1.608106 Accuracy: 22.01% 27 Validation loss: 1.612873 Best loss: 1.608106 Accuracy: 22.01% 28 Validation loss: 1.622145 Best loss: 1.608106 Accuracy: 19.08% 29 Validation loss: 1.609206 Best loss: 1.608106 Accuracy: 22.01% 30 Validation loss: 1.607765 Best loss: 1.607765 Accuracy: 20.91% 31 Validation loss: 1.613588 Best loss: 1.607765 Accuracy: 18.73% 32 Validation loss: 1.609266 Best loss: 1.607765 Accuracy: 22.01% 33 Validation loss: 1.609563 Best loss: 1.607765 Accuracy: 22.01% 34 Validation loss: 1.610188 Best loss: 1.607765 Accuracy: 22.01% 35 Validation loss: 1.609365 Best loss: 1.607765 Accuracy: 22.01% 36 Validation loss: 1.610132 Best loss: 1.607765 Accuracy: 19.27% 37 Validation loss: 1.612202 Best loss: 1.607765 Accuracy: 22.01% 38 Validation loss: 1.612038 Best loss: 1.607765 Accuracy: 22.01% 39 Validation loss: 1.614102 Best loss: 1.607765 Accuracy: 19.27% 40 Validation loss: 1.608473 Best loss: 1.607765 Accuracy: 22.01% 41 Validation loss: 1.609536 Best loss: 1.607765 Accuracy: 22.01% 42 Validation loss: 1.610706 Best loss: 1.607765 Accuracy: 19.27% 43 Validation loss: 1.614007 Best loss: 1.607765 Accuracy: 22.01% 44 Validation loss: 1.616754 Best loss: 1.607765 Accuracy: 22.01% 45 Validation loss: 1.610697 Best loss: 1.607765 Accuracy: 22.01% 46 Validation loss: 1.612055 Best loss: 1.607765 Accuracy: 18.73% 47 Validation loss: 1.608928 Best loss: 1.607765 Accuracy: 22.01% 48 Validation loss: 1.608292 Best loss: 1.607765 Accuracy: 22.01% 49 Validation loss: 1.617473 Best loss: 1.607765 Accuracy: 19.27% 50 Validation loss: 1.612356 Best loss: 1.607765 Accuracy: 19.27% 51 Validation loss: 1.610517 Best loss: 1.607765 Accuracy: 19.08% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 4.6min [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5 0 Validation loss: 1.615278 Best loss: 1.615278 Accuracy: 22.01% 1 Validation loss: 1.613000 Best loss: 1.613000 Accuracy: 19.27% 2 Validation loss: 1.613225 Best loss: 1.613000 Accuracy: 19.27% 3 Validation loss: 1.618716 Best loss: 1.613000 Accuracy: 22.01% 4 Validation loss: 1.608358 Best loss: 1.608358 Accuracy: 22.01% 5 Validation loss: 1.612527 Best loss: 1.608358 Accuracy: 22.01% 6 Validation loss: 1.613944 Best loss: 1.608358 Accuracy: 19.27% 7 Validation loss: 1.612977 Best loss: 1.608358 Accuracy: 22.01% 8 Validation loss: 1.611469 Best loss: 1.608358 Accuracy: 22.01% 9 Validation loss: 1.610273 Best loss: 1.608358 Accuracy: 22.01% 10 Validation loss: 1.610118 Best loss: 1.608358 Accuracy: 22.01% 11 Validation loss: 1.615040 Best loss: 1.608358 Accuracy: 22.01% 12 Validation loss: 1.609143 Best loss: 1.608358 Accuracy: 20.91% 13 Validation loss: 1.612667 Best loss: 1.608358 Accuracy: 19.27% 14 Validation loss: 1.610130 Best loss: 1.608358 Accuracy: 20.91% 15 Validation loss: 1.611327 Best loss: 1.608358 Accuracy: 22.01% 16 Validation loss: 1.620624 Best loss: 1.608358 Accuracy: 22.01% 17 Validation loss: 1.614154 Best loss: 1.608358 Accuracy: 19.08% 18 Validation loss: 1.607982 Best loss: 1.607982 Accuracy: 22.01% 19 Validation loss: 1.610182 Best loss: 1.607982 Accuracy: 19.08% 20 Validation loss: 1.612591 Best loss: 1.607982 Accuracy: 19.27% 21 Validation loss: 1.620058 Best loss: 1.607982 Accuracy: 22.01% 22 Validation loss: 1.611555 Best loss: 1.607982 Accuracy: 22.01% 23 Validation loss: 1.614218 Best loss: 1.607982 Accuracy: 22.01% 24 Validation loss: 1.612510 Best loss: 1.607982 Accuracy: 20.91% 25 Validation loss: 1.609958 Best loss: 1.607982 Accuracy: 22.01% 26 Validation loss: 1.607882 Best loss: 1.607882 Accuracy: 22.01% 27 Validation loss: 1.610300 Best loss: 1.607882 Accuracy: 22.01% 28 Validation loss: 1.617167 Best loss: 1.607882 Accuracy: 19.08% 29 Validation loss: 1.610925 Best loss: 1.607882 Accuracy: 22.01% 30 Validation loss: 1.609472 Best loss: 1.607882 Accuracy: 19.08% 31 Validation loss: 1.609234 Best loss: 1.607882 Accuracy: 22.01% 32 Validation loss: 1.609479 Best loss: 1.607882 Accuracy: 22.01% 33 Validation loss: 1.609797 Best loss: 1.607882 Accuracy: 22.01% 34 Validation loss: 1.609309 Best loss: 1.607882 Accuracy: 22.01% 35 Validation loss: 1.611006 Best loss: 1.607882 Accuracy: 22.01% 36 Validation loss: 1.610507 Best loss: 1.607882 Accuracy: 19.27% 37 Validation loss: 1.608582 Best loss: 1.607882 Accuracy: 22.01% 38 Validation loss: 1.612431 Best loss: 1.607882 Accuracy: 22.01% 39 Validation loss: 1.616542 Best loss: 1.607882 Accuracy: 19.27% 40 Validation loss: 1.610356 Best loss: 1.607882 Accuracy: 19.08% 41 Validation loss: 1.613422 Best loss: 1.607882 Accuracy: 19.08% 42 Validation loss: 1.610970 Best loss: 1.607882 Accuracy: 22.01% 43 Validation loss: 1.609564 Best loss: 1.607882 Accuracy: 20.91% 44 Validation loss: 1.612211 Best loss: 1.607882 Accuracy: 22.01% 45 Validation loss: 1.609153 Best loss: 1.607882 Accuracy: 22.01% 46 Validation loss: 1.608850 Best loss: 1.607882 Accuracy: 22.01% 47 Validation loss: 1.625009 Best loss: 1.607882 Accuracy: 19.27% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=140, batch_size=10, learning_rate=0.02, dropout_rate=0.5, total= 4.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 7.291085 Best loss: 7.291085 Accuracy: 38.12% 1 Validation loss: 13.960744 Best loss: 7.291085 Accuracy: 27.56% 2 Validation loss: 143.800461 Best loss: 7.291085 Accuracy: 24.35% 3 Validation loss: 53.961720 Best loss: 7.291085 Accuracy: 26.08% 4 Validation loss: 177.890961 Best loss: 7.291085 Accuracy: 34.64% 5 Validation loss: 982.010925 Best loss: 7.291085 Accuracy: 38.86% 6 Validation loss: 149.227783 Best loss: 7.291085 Accuracy: 39.52% 7 Validation loss: 50.282032 Best loss: 7.291085 Accuracy: 40.66% 8 Validation loss: 42.346588 Best loss: 7.291085 Accuracy: 39.64% 9 Validation loss: 78.049774 Best loss: 7.291085 Accuracy: 36.59% 10 Validation loss: 85.312828 Best loss: 7.291085 Accuracy: 39.48% 11 Validation loss: 140.767075 Best loss: 7.291085 Accuracy: 40.62% 12 Validation loss: 166.303284 Best loss: 7.291085 Accuracy: 38.78% 13 Validation loss: 231.368179 Best loss: 7.291085 Accuracy: 37.76% 14 Validation loss: 469.876862 Best loss: 7.291085 Accuracy: 40.97% 15 Validation loss: 157.605484 Best loss: 7.291085 Accuracy: 38.31% 16 Validation loss: 102.166634 Best loss: 7.291085 Accuracy: 37.10% 17 Validation loss: 67.468765 Best loss: 7.291085 Accuracy: 33.35% 18 Validation loss: 117.542450 Best loss: 7.291085 Accuracy: 39.09% 19 Validation loss: 2993.800537 Best loss: 7.291085 Accuracy: 33.74% 20 Validation loss: 125.408363 Best loss: 7.291085 Accuracy: 30.69% 21 Validation loss: 341.019226 Best loss: 7.291085 Accuracy: 35.89% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 14.117893 Best loss: 14.117893 Accuracy: 33.03% 1 Validation loss: 72.446373 Best loss: 14.117893 Accuracy: 22.95% 2 Validation loss: 25.529039 Best loss: 14.117893 Accuracy: 21.31% 3 Validation loss: 15.194607 Best loss: 14.117893 Accuracy: 40.62% 4 Validation loss: 128.600510 Best loss: 14.117893 Accuracy: 30.06% 5 Validation loss: 58.148781 Best loss: 14.117893 Accuracy: 33.78% 6 Validation loss: 42.588669 Best loss: 14.117893 Accuracy: 36.20% 7 Validation loss: 45.164001 Best loss: 14.117893 Accuracy: 36.24% 8 Validation loss: 47.177147 Best loss: 14.117893 Accuracy: 36.43% 9 Validation loss: 145.526047 Best loss: 14.117893 Accuracy: 36.67% 10 Validation loss: 156.947952 Best loss: 14.117893 Accuracy: 34.52% 11 Validation loss: 230.246552 Best loss: 14.117893 Accuracy: 42.53% 12 Validation loss: 71.782562 Best loss: 14.117893 Accuracy: 34.68% 13 Validation loss: 65.238464 Best loss: 14.117893 Accuracy: 43.78% 14 Validation loss: 147.587280 Best loss: 14.117893 Accuracy: 36.86% 15 Validation loss: 198.662170 Best loss: 14.117893 Accuracy: 32.56% 16 Validation loss: 49.564671 Best loss: 14.117893 Accuracy: 33.70% 17 Validation loss: 96.058838 Best loss: 14.117893 Accuracy: 33.42% 18 Validation loss: 335.193207 Best loss: 14.117893 Accuracy: 37.57% 19 Validation loss: 373.975067 Best loss: 14.117893 Accuracy: 34.56% 20 Validation loss: 238.437897 Best loss: 14.117893 Accuracy: 30.88% 21 Validation loss: 116.621544 Best loss: 14.117893 Accuracy: 32.41% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5, total= 2.2min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 13.098383 Best loss: 13.098383 Accuracy: 47.03% 1 Validation loss: 4.614475 Best loss: 4.614475 Accuracy: 50.23% 2 Validation loss: 17.803417 Best loss: 4.614475 Accuracy: 41.09% 3 Validation loss: 437.555603 Best loss: 4.614475 Accuracy: 39.64% 4 Validation loss: 391.955322 Best loss: 4.614475 Accuracy: 36.47% 5 Validation loss: 63.099213 Best loss: 4.614475 Accuracy: 39.17% 6 Validation loss: 661.921082 Best loss: 4.614475 Accuracy: 34.17% 7 Validation loss: 163.844040 Best loss: 4.614475 Accuracy: 38.70% 8 Validation loss: 66.511475 Best loss: 4.614475 Accuracy: 38.78% 9 Validation loss: 44.068741 Best loss: 4.614475 Accuracy: 40.19% 10 Validation loss: 49.115116 Best loss: 4.614475 Accuracy: 44.64% 11 Validation loss: 269.080780 Best loss: 4.614475 Accuracy: 51.68% 12 Validation loss: 67.941986 Best loss: 4.614475 Accuracy: 52.50% 13 Validation loss: 172.690613 Best loss: 4.614475 Accuracy: 54.69% 14 Validation loss: 128.433395 Best loss: 4.614475 Accuracy: 53.09% 15 Validation loss: 502.366608 Best loss: 4.614475 Accuracy: 38.98% 16 Validation loss: 52.698982 Best loss: 4.614475 Accuracy: 56.96% 17 Validation loss: 157.043472 Best loss: 4.614475 Accuracy: 52.70% 18 Validation loss: 70.356606 Best loss: 4.614475 Accuracy: 55.98% 19 Validation loss: 447.603546 Best loss: 4.614475 Accuracy: 47.97% 20 Validation loss: 396.315094 Best loss: 4.614475 Accuracy: 53.36% 21 Validation loss: 90.841278 Best loss: 4.614475 Accuracy: 55.82% 22 Validation loss: 125.049232 Best loss: 4.614475 Accuracy: 56.57% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=140, batch_size=10, learning_rate=0.01, dropout_rate=0.5, total= 2.3min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3 0 Validation loss: 383.280304 Best loss: 383.280304 Accuracy: 18.73% 1 Validation loss: 24.638739 Best loss: 24.638739 Accuracy: 23.73% 2 Validation loss: 8.675844 Best loss: 8.675844 Accuracy: 42.77% 3 Validation loss: 7.063754 Best loss: 7.063754 Accuracy: 36.86% 4 Validation loss: 18.780939 Best loss: 7.063754 Accuracy: 36.20% 5 Validation loss: 7.737678 Best loss: 7.063754 Accuracy: 46.17% 6 Validation loss: 11.837118 Best loss: 7.063754 Accuracy: 46.60% 7 Validation loss: 0.886742 Best loss: 0.886742 Accuracy: 79.71% 8 Validation loss: 1277.964966 Best loss: 0.886742 Accuracy: 19.08% 9 Validation loss: 14370989.000000 Best loss: 0.886742 Accuracy: 17.01% 10 Validation loss: 137359.515625 Best loss: 0.886742 Accuracy: 48.55% 11 Validation loss: 73522.437500 Best loss: 0.886742 Accuracy: 44.61% 12 Validation loss: 78010.867188 Best loss: 0.886742 Accuracy: 35.65% 13 Validation loss: 105453.046875 Best loss: 0.886742 Accuracy: 38.74% 14 Validation loss: 22114.488281 Best loss: 0.886742 Accuracy: 49.49% 15 Validation loss: 79813.414062 Best loss: 0.886742 Accuracy: 38.12% 16 Validation loss: 65781.312500 Best loss: 0.886742 Accuracy: 40.34% 17 Validation loss: 108375.804688 Best loss: 0.886742 Accuracy: 33.97% 18 Validation loss: 65381.835938 Best loss: 0.886742 Accuracy: 34.09% 19 Validation loss: 77366.000000 Best loss: 0.886742 Accuracy: 34.17% 20 Validation loss: 73836.164062 Best loss: 0.886742 Accuracy: 37.65% 21 Validation loss: 65648.726562 Best loss: 0.886742 Accuracy: 46.83% 22 Validation loss: 48453.097656 Best loss: 0.886742 Accuracy: 50.39% 23 Validation loss: 17737.771484 Best loss: 0.886742 Accuracy: 48.40% 24 Validation loss: 51797.652344 Best loss: 0.886742 Accuracy: 36.04% 25 Validation loss: 59550.074219 Best loss: 0.886742 Accuracy: 36.32% 26 Validation loss: 40949.308594 Best loss: 0.886742 Accuracy: 44.68% 27 Validation loss: 42677.121094 Best loss: 0.886742 Accuracy: 36.59% 28 Validation loss: 48415.968750 Best loss: 0.886742 Accuracy: 53.09% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3, total= 6.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3 0 Validation loss: 533.093994 Best loss: 533.093994 Accuracy: 22.01% 1 Validation loss: 36.845993 Best loss: 36.845993 Accuracy: 43.94% 2 Validation loss: 33.863213 Best loss: 33.863213 Accuracy: 39.01% 3 Validation loss: 37.866077 Best loss: 33.863213 Accuracy: 42.53% 4 Validation loss: 5.672101 Best loss: 5.672101 Accuracy: 52.54% 5 Validation loss: 7.941962 Best loss: 5.672101 Accuracy: 59.81% 6 Validation loss: 7.554715 Best loss: 5.672101 Accuracy: 64.93% 7 Validation loss: 2.110953 Best loss: 2.110953 Accuracy: 72.83% 8 Validation loss: 1.498095 Best loss: 1.498095 Accuracy: 67.71% 9 Validation loss: 7.432526 Best loss: 1.498095 Accuracy: 53.67% 10 Validation loss: 2.786276 Best loss: 1.498095 Accuracy: 66.97% 11 Validation loss: 1.171038 Best loss: 1.171038 Accuracy: 81.00% 12 Validation loss: 4.360723 Best loss: 1.171038 Accuracy: 65.95% 13 Validation loss: 3.560621 Best loss: 1.171038 Accuracy: 71.81% 14 Validation loss: 1.434296 Best loss: 1.171038 Accuracy: 74.35% 15 Validation loss: 3.781398 Best loss: 1.171038 Accuracy: 72.75% 16 Validation loss: 2.286085 Best loss: 1.171038 Accuracy: 73.34% 17 Validation loss: 3.023535 Best loss: 1.171038 Accuracy: 73.73% 18 Validation loss: 1.390475 Best loss: 1.171038 Accuracy: 73.61% 19 Validation loss: 0.903162 Best loss: 0.903162 Accuracy: 79.52% 20 Validation loss: 2.103949 Best loss: 0.903162 Accuracy: 74.08% 21 Validation loss: 0.643745 Best loss: 0.643745 Accuracy: 80.34% 22 Validation loss: 1.730224 Best loss: 0.643745 Accuracy: 75.33% 23 Validation loss: 1.246808 Best loss: 0.643745 Accuracy: 76.00% 24 Validation loss: 0.572442 Best loss: 0.572442 Accuracy: 81.78% 25 Validation loss: 0.857622 Best loss: 0.572442 Accuracy: 78.89% 26 Validation loss: 0.436042 Best loss: 0.436042 Accuracy: 90.07% 27 Validation loss: 0.565692 Best loss: 0.436042 Accuracy: 87.18% 28 Validation loss: 1.032447 Best loss: 0.436042 Accuracy: 75.88% 29 Validation loss: 0.581612 Best loss: 0.436042 Accuracy: 84.09% 30 Validation loss: 0.420736 Best loss: 0.420736 Accuracy: 91.28% 31 Validation loss: 0.428374 Best loss: 0.420736 Accuracy: 91.32% 32 Validation loss: 0.392866 Best loss: 0.392866 Accuracy: 90.58% 33 Validation loss: 0.297680 Best loss: 0.297680 Accuracy: 92.61% 34 Validation loss: 1.110434 Best loss: 0.297680 Accuracy: 91.63% 35 Validation loss: 0.760531 Best loss: 0.297680 Accuracy: 91.79% 36 Validation loss: 0.785910 Best loss: 0.297680 Accuracy: 88.00% 37 Validation loss: 0.470498 Best loss: 0.297680 Accuracy: 91.40% 38 Validation loss: 0.670901 Best loss: 0.297680 Accuracy: 86.83% 39 Validation loss: 0.413603 Best loss: 0.297680 Accuracy: 92.53% 40 Validation loss: 0.322472 Best loss: 0.297680 Accuracy: 93.35% 41 Validation loss: 0.402028 Best loss: 0.297680 Accuracy: 92.53% 42 Validation loss: 0.470635 Best loss: 0.297680 Accuracy: 89.68% 43 Validation loss: 0.541670 Best loss: 0.297680 Accuracy: 91.09% 44 Validation loss: 0.857423 Best loss: 0.297680 Accuracy: 91.36% 45 Validation loss: 0.670130 Best loss: 0.297680 Accuracy: 87.18% 46 Validation loss: 0.712324 Best loss: 0.297680 Accuracy: 87.06% 47 Validation loss: 0.531153 Best loss: 0.297680 Accuracy: 90.19% 48 Validation loss: 0.322306 Best loss: 0.297680 Accuracy: 94.06% 49 Validation loss: 0.402871 Best loss: 0.297680 Accuracy: 91.52% 50 Validation loss: 0.273669 Best loss: 0.273669 Accuracy: 93.82% 51 Validation loss: 0.296457 Best loss: 0.273669 Accuracy: 92.89% 52 Validation loss: 0.256905 Best loss: 0.256905 Accuracy: 92.96% 53 Validation loss: 1.770828 Best loss: 0.256905 Accuracy: 69.78% 54 Validation loss: 1.128683 Best loss: 0.256905 Accuracy: 92.85% 55 Validation loss: 0.835562 Best loss: 0.256905 Accuracy: 91.83% 56 Validation loss: 0.625701 Best loss: 0.256905 Accuracy: 92.06% 57 Validation loss: 0.531788 Best loss: 0.256905 Accuracy: 93.59% 58 Validation loss: 0.325365 Best loss: 0.256905 Accuracy: 93.63% 59 Validation loss: 0.360797 Best loss: 0.256905 Accuracy: 92.69% 60 Validation loss: 0.247998 Best loss: 0.247998 Accuracy: 94.49% 61 Validation loss: 0.322240 Best loss: 0.247998 Accuracy: 92.30% 62 Validation loss: 0.246837 Best loss: 0.246837 Accuracy: 93.75% 63 Validation loss: 0.263004 Best loss: 0.246837 Accuracy: 94.96% 64 Validation loss: 0.314265 Best loss: 0.246837 Accuracy: 92.49% 65 Validation loss: 0.322100 Best loss: 0.246837 Accuracy: 92.77% 66 Validation loss: 0.267447 Best loss: 0.246837 Accuracy: 94.06% 67 Validation loss: 0.259410 Best loss: 0.246837 Accuracy: 93.67% 68 Validation loss: 0.273342 Best loss: 0.246837 Accuracy: 93.51% 69 Validation loss: 0.357491 Best loss: 0.246837 Accuracy: 92.14% 70 Validation loss: 0.336809 Best loss: 0.246837 Accuracy: 92.34% 71 Validation loss: 0.231866 Best loss: 0.231866 Accuracy: 95.19% 72 Validation loss: 0.224675 Best loss: 0.224675 Accuracy: 93.94% 73 Validation loss: 0.258299 Best loss: 0.224675 Accuracy: 95.62% 74 Validation loss: 0.269766 Best loss: 0.224675 Accuracy: 92.73% 75 Validation loss: 0.305989 Best loss: 0.224675 Accuracy: 93.04% 76 Validation loss: 0.294764 Best loss: 0.224675 Accuracy: 92.49% 77 Validation loss: 0.237500 Best loss: 0.224675 Accuracy: 94.10% 78 Validation loss: 0.249720 Best loss: 0.224675 Accuracy: 94.10% 79 Validation loss: 0.200424 Best loss: 0.200424 Accuracy: 95.07% 80 Validation loss: 0.222507 Best loss: 0.200424 Accuracy: 94.10% 81 Validation loss: 0.294542 Best loss: 0.200424 Accuracy: 93.47% 82 Validation loss: 0.249577 Best loss: 0.200424 Accuracy: 95.27% 83 Validation loss: 0.218369 Best loss: 0.200424 Accuracy: 95.11% 84 Validation loss: 1.238165 Best loss: 0.200424 Accuracy: 89.44% 85 Validation loss: 0.474255 Best loss: 0.200424 Accuracy: 91.91% 86 Validation loss: 0.444071 Best loss: 0.200424 Accuracy: 92.18% 87 Validation loss: 0.382921 Best loss: 0.200424 Accuracy: 93.47% 88 Validation loss: 0.257497 Best loss: 0.200424 Accuracy: 95.35% 89 Validation loss: 0.239158 Best loss: 0.200424 Accuracy: 94.21% 90 Validation loss: 0.255365 Best loss: 0.200424 Accuracy: 93.75% 91 Validation loss: 0.224886 Best loss: 0.200424 Accuracy: 94.18% 92 Validation loss: 0.241405 Best loss: 0.200424 Accuracy: 93.98% 93 Validation loss: 0.239400 Best loss: 0.200424 Accuracy: 96.01% 94 Validation loss: 0.200082 Best loss: 0.200082 Accuracy: 95.47% 95 Validation loss: 0.206659 Best loss: 0.200082 Accuracy: 95.15% 96 Validation loss: 0.203379 Best loss: 0.200082 Accuracy: 94.68% 97 Validation loss: 0.275086 Best loss: 0.200082 Accuracy: 94.64% 98 Validation loss: 0.409321 Best loss: 0.200082 Accuracy: 94.10% 99 Validation loss: 0.300940 Best loss: 0.200082 Accuracy: 92.96% 100 Validation loss: 0.278322 Best loss: 0.200082 Accuracy: 95.54% 101 Validation loss: 0.247456 Best loss: 0.200082 Accuracy: 95.35% 102 Validation loss: 0.196384 Best loss: 0.196384 Accuracy: 95.47% 103 Validation loss: 0.226481 Best loss: 0.196384 Accuracy: 95.23% 104 Validation loss: 0.319343 Best loss: 0.196384 Accuracy: 94.45% 105 Validation loss: 0.229366 Best loss: 0.196384 Accuracy: 95.31% 106 Validation loss: 0.205862 Best loss: 0.196384 Accuracy: 95.78% 107 Validation loss: 0.244694 Best loss: 0.196384 Accuracy: 95.04% 108 Validation loss: 0.299419 Best loss: 0.196384 Accuracy: 94.41% 109 Validation loss: 0.262832 Best loss: 0.196384 Accuracy: 94.06% 110 Validation loss: 0.191804 Best loss: 0.191804 Accuracy: 95.58% 111 Validation loss: 0.210752 Best loss: 0.191804 Accuracy: 94.49% 112 Validation loss: 0.291964 Best loss: 0.191804 Accuracy: 94.84% 113 Validation loss: 176174000.000000 Best loss: 0.191804 Accuracy: 6.61% 114 Validation loss: 321438336.000000 Best loss: 0.191804 Accuracy: 18.73% 115 Validation loss: 5874134.000000 Best loss: 0.191804 Accuracy: 59.58% 116 Validation loss: 20427492.000000 Best loss: 0.191804 Accuracy: 34.95% 117 Validation loss: 1161222.375000 Best loss: 0.191804 Accuracy: 65.32% 118 Validation loss: 1184599.625000 Best loss: 0.191804 Accuracy: 67.90% 119 Validation loss: 8724494.000000 Best loss: 0.191804 Accuracy: 41.28% 120 Validation loss: 941970.437500 Best loss: 0.191804 Accuracy: 56.88% 121 Validation loss: 640625.062500 Best loss: 0.191804 Accuracy: 58.05% 122 Validation loss: 334382.281250 Best loss: 0.191804 Accuracy: 66.77% 123 Validation loss: 587904.625000 Best loss: 0.191804 Accuracy: 66.69% 124 Validation loss: 957861.750000 Best loss: 0.191804 Accuracy: 58.72% 125 Validation loss: 1405744.500000 Best loss: 0.191804 Accuracy: 65.25% 126 Validation loss: 1657456.375000 Best loss: 0.191804 Accuracy: 66.69% 127 Validation loss: 429488.531250 Best loss: 0.191804 Accuracy: 63.02% 128 Validation loss: 313032.062500 Best loss: 0.191804 Accuracy: 60.16% 129 Validation loss: 1555927.125000 Best loss: 0.191804 Accuracy: 53.56% 130 Validation loss: 593124.500000 Best loss: 0.191804 Accuracy: 70.17% 131 Validation loss: 435470.812500 Best loss: 0.191804 Accuracy: 54.93% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3, total= 23.4s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3 0 Validation loss: 1240.660522 Best loss: 1240.660522 Accuracy: 26.94% 1 Validation loss: 65.158707 Best loss: 65.158707 Accuracy: 30.88% 2 Validation loss: 61.589401 Best loss: 61.589401 Accuracy: 27.09% 3 Validation loss: 10.376982 Best loss: 10.376982 Accuracy: 47.97% 4 Validation loss: 6.978062 Best loss: 6.978062 Accuracy: 49.49% 5 Validation loss: 7.762548 Best loss: 6.978062 Accuracy: 56.02% 6 Validation loss: 18.304737 Best loss: 6.978062 Accuracy: 50.90% 7 Validation loss: 2.082688 Best loss: 2.082688 Accuracy: 77.64% 8 Validation loss: 3.854478 Best loss: 2.082688 Accuracy: 64.31% 9 Validation loss: 2.770405 Best loss: 2.082688 Accuracy: 69.70% 10 Validation loss: 22.579556 Best loss: 2.082688 Accuracy: 28.85% 11 Validation loss: 4.771555 Best loss: 2.082688 Accuracy: 57.43% 12 Validation loss: 1.690885 Best loss: 1.690885 Accuracy: 70.02% 13 Validation loss: 2.372721 Best loss: 1.690885 Accuracy: 70.33% 14 Validation loss: 2.366341 Best loss: 1.690885 Accuracy: 69.90% 15 Validation loss: 2.524565 Best loss: 1.690885 Accuracy: 63.72% 16 Validation loss: 2.545362 Best loss: 1.690885 Accuracy: 71.07% 17 Validation loss: 1.809055 Best loss: 1.690885 Accuracy: 72.24% 18 Validation loss: 0.952319 Best loss: 0.952319 Accuracy: 82.10% 19 Validation loss: 1.325912 Best loss: 0.952319 Accuracy: 69.31% 20 Validation loss: 1.162882 Best loss: 0.952319 Accuracy: 75.76% 21 Validation loss: 0.923844 Best loss: 0.923844 Accuracy: 82.21% 22 Validation loss: 0.901744 Best loss: 0.901744 Accuracy: 76.35% 23 Validation loss: 0.652234 Best loss: 0.652234 Accuracy: 85.65% 24 Validation loss: 0.713460 Best loss: 0.652234 Accuracy: 88.15% 25 Validation loss: 0.926021 Best loss: 0.652234 Accuracy: 82.37% 26 Validation loss: 1.361670 Best loss: 0.652234 Accuracy: 71.70% 27 Validation loss: 2.661212 Best loss: 0.652234 Accuracy: 68.37% 28 Validation loss: 1.783680 Best loss: 0.652234 Accuracy: 67.63% 29 Validation loss: 1.252522 Best loss: 0.652234 Accuracy: 81.12% 30 Validation loss: 1.307158 Best loss: 0.652234 Accuracy: 87.92% 31 Validation loss: 1.330288 Best loss: 0.652234 Accuracy: 72.56% 32 Validation loss: 0.690191 Best loss: 0.652234 Accuracy: 87.88% 33 Validation loss: 0.598684 Best loss: 0.598684 Accuracy: 89.05% 34 Validation loss: 1.245372 Best loss: 0.598684 Accuracy: 78.50% 35 Validation loss: 0.414936 Best loss: 0.414936 Accuracy: 90.11% 36 Validation loss: 0.459666 Best loss: 0.414936 Accuracy: 85.26% 37 Validation loss: 27036438.000000 Best loss: 0.414936 Accuracy: 28.54% 38 Validation loss: 2092202.875000 Best loss: 0.414936 Accuracy: 32.99% 39 Validation loss: 1220951.125000 Best loss: 0.414936 Accuracy: 45.19% 40 Validation loss: 548216.375000 Best loss: 0.414936 Accuracy: 54.65% 41 Validation loss: 382578.781250 Best loss: 0.414936 Accuracy: 49.22% 42 Validation loss: 363484.218750 Best loss: 0.414936 Accuracy: 56.61% 43 Validation loss: 219770.468750 Best loss: 0.414936 Accuracy: 63.33% 44 Validation loss: 192933.671875 Best loss: 0.414936 Accuracy: 54.42% 45 Validation loss: 255998.093750 Best loss: 0.414936 Accuracy: 46.40% 46 Validation loss: 56198.742188 Best loss: 0.414936 Accuracy: 64.23% 47 Validation loss: 469690.500000 Best loss: 0.414936 Accuracy: 49.84% 48 Validation loss: 26949.705078 Best loss: 0.414936 Accuracy: 81.20% 49 Validation loss: 65965.820312 Best loss: 0.414936 Accuracy: 75.22% 50 Validation loss: 98802.585938 Best loss: 0.414936 Accuracy: 74.55% 51 Validation loss: 70784.828125 Best loss: 0.414936 Accuracy: 58.21% 52 Validation loss: 106448.148438 Best loss: 0.414936 Accuracy: 74.08% 53 Validation loss: 363219.843750 Best loss: 0.414936 Accuracy: 61.26% 54 Validation loss: 155356.328125 Best loss: 0.414936 Accuracy: 56.29% 55 Validation loss: 140781.765625 Best loss: 0.414936 Accuracy: 78.30% 56 Validation loss: 56925.441406 Best loss: 0.414936 Accuracy: 82.99% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=160, batch_size=500, learning_rate=0.1, dropout_rate=0.3, total= 10.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.166688 Best loss: 0.166688 Accuracy: 95.35% 1 Validation loss: 0.156792 Best loss: 0.156792 Accuracy: 95.62% 2 Validation loss: 0.176655 Best loss: 0.156792 Accuracy: 95.70% 3 Validation loss: 0.206907 Best loss: 0.156792 Accuracy: 94.80% 4 Validation loss: 0.253221 Best loss: 0.156792 Accuracy: 90.77% 5 Validation loss: 0.449601 Best loss: 0.156792 Accuracy: 85.03% 6 Validation loss: 0.520023 Best loss: 0.156792 Accuracy: 81.70% 7 Validation loss: 0.498231 Best loss: 0.156792 Accuracy: 80.06% 8 Validation loss: 0.605225 Best loss: 0.156792 Accuracy: 81.39% 9 Validation loss: 0.420640 Best loss: 0.156792 Accuracy: 86.67% 10 Validation loss: 0.367251 Best loss: 0.156792 Accuracy: 90.03% 11 Validation loss: 0.658575 Best loss: 0.156792 Accuracy: 74.39% 12 Validation loss: 0.806388 Best loss: 0.156792 Accuracy: 73.85% 13 Validation loss: 0.991039 Best loss: 0.156792 Accuracy: 60.63% 14 Validation loss: 0.983025 Best loss: 0.156792 Accuracy: 58.09% 15 Validation loss: 1.105367 Best loss: 0.156792 Accuracy: 51.17% 16 Validation loss: 1.026881 Best loss: 0.156792 Accuracy: 55.86% 17 Validation loss: 1.085987 Best loss: 0.156792 Accuracy: 52.31% 18 Validation loss: 0.992540 Best loss: 0.156792 Accuracy: 57.62% 19 Validation loss: 1.277528 Best loss: 0.156792 Accuracy: 47.19% 20 Validation loss: 1.203295 Best loss: 0.156792 Accuracy: 48.48% 21 Validation loss: 1.161827 Best loss: 0.156792 Accuracy: 51.45% 22 Validation loss: 0.857447 Best loss: 0.156792 Accuracy: 61.49% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 16.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.184707 Best loss: 0.184707 Accuracy: 95.11% 1 Validation loss: 0.164625 Best loss: 0.164625 Accuracy: 95.66% 2 Validation loss: 0.198874 Best loss: 0.164625 Accuracy: 93.28% 3 Validation loss: 0.264529 Best loss: 0.164625 Accuracy: 92.53% 4 Validation loss: 0.624021 Best loss: 0.164625 Accuracy: 82.80% 5 Validation loss: 0.416559 Best loss: 0.164625 Accuracy: 89.21% 6 Validation loss: 0.397295 Best loss: 0.164625 Accuracy: 83.27% 7 Validation loss: 0.305557 Best loss: 0.164625 Accuracy: 90.89% 8 Validation loss: 0.334356 Best loss: 0.164625 Accuracy: 89.09% 9 Validation loss: 0.452521 Best loss: 0.164625 Accuracy: 84.48% 10 Validation loss: 0.540186 Best loss: 0.164625 Accuracy: 76.62% 11 Validation loss: 0.515674 Best loss: 0.164625 Accuracy: 79.40% 12 Validation loss: 0.484123 Best loss: 0.164625 Accuracy: 80.77% 13 Validation loss: 0.368373 Best loss: 0.164625 Accuracy: 87.45% 14 Validation loss: 0.376275 Best loss: 0.164625 Accuracy: 85.46% 15 Validation loss: 0.443792 Best loss: 0.164625 Accuracy: 82.56% 16 Validation loss: 0.580366 Best loss: 0.164625 Accuracy: 74.86% 17 Validation loss: 0.440899 Best loss: 0.164625 Accuracy: 83.39% 18 Validation loss: 0.431376 Best loss: 0.164625 Accuracy: 84.68% 19 Validation loss: 0.385098 Best loss: 0.164625 Accuracy: 84.01% 20 Validation loss: 0.398682 Best loss: 0.164625 Accuracy: 82.13% 21 Validation loss: 0.544260 Best loss: 0.164625 Accuracy: 81.55% 22 Validation loss: 0.449771 Best loss: 0.164625 Accuracy: 86.43% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 15.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.197585 Best loss: 0.197585 Accuracy: 94.68% 1 Validation loss: 0.180750 Best loss: 0.180750 Accuracy: 94.41% 2 Validation loss: 0.168394 Best loss: 0.168394 Accuracy: 94.96% 3 Validation loss: 0.268909 Best loss: 0.168394 Accuracy: 95.35% 4 Validation loss: 0.334720 Best loss: 0.168394 Accuracy: 90.15% 5 Validation loss: 0.490688 Best loss: 0.168394 Accuracy: 82.99% 6 Validation loss: 0.506413 Best loss: 0.168394 Accuracy: 79.75% 7 Validation loss: 0.671481 Best loss: 0.168394 Accuracy: 71.31% 8 Validation loss: 0.502159 Best loss: 0.168394 Accuracy: 81.94% 9 Validation loss: 0.458503 Best loss: 0.168394 Accuracy: 82.68% 10 Validation loss: 0.492241 Best loss: 0.168394 Accuracy: 79.79% 11 Validation loss: 0.379895 Best loss: 0.168394 Accuracy: 83.42% 12 Validation loss: 0.343954 Best loss: 0.168394 Accuracy: 88.08% 13 Validation loss: 1.937891 Best loss: 0.168394 Accuracy: 41.13% 14 Validation loss: 1.894984 Best loss: 0.168394 Accuracy: 31.16% 15 Validation loss: 1.441838 Best loss: 0.168394 Accuracy: 35.65% 16 Validation loss: 0.963899 Best loss: 0.168394 Accuracy: 55.86% 17 Validation loss: 1.555002 Best loss: 0.168394 Accuracy: 50.86% 18 Validation loss: 1.182890 Best loss: 0.168394 Accuracy: 51.76% 19 Validation loss: 1.301411 Best loss: 0.168394 Accuracy: 38.58% 20 Validation loss: 1.786113 Best loss: 0.168394 Accuracy: 36.63% 21 Validation loss: 1.000882 Best loss: 0.168394 Accuracy: 55.98% 22 Validation loss: 0.863050 Best loss: 0.168394 Accuracy: 60.44% 23 Validation loss: 0.714390 Best loss: 0.168394 Accuracy: 70.68% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.01, dropout_rate=0.6, total= 16.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 34468.972656 Best loss: 34468.972656 Accuracy: 18.73% 1 Validation loss: 59109.640625 Best loss: 34468.972656 Accuracy: 24.24% 2 Validation loss: 16980.435547 Best loss: 16980.435547 Accuracy: 19.08% 3 Validation loss: 69559.156250 Best loss: 16980.435547 Accuracy: 18.73% 4 Validation loss: 192194.000000 Best loss: 16980.435547 Accuracy: 18.73% 5 Validation loss: 780984.750000 Best loss: 16980.435547 Accuracy: 19.08% 6 Validation loss: 420355.593750 Best loss: 16980.435547 Accuracy: 19.31% 7 Validation loss: 1008517.875000 Best loss: 16980.435547 Accuracy: 19.27% 8 Validation loss: 2140630.250000 Best loss: 16980.435547 Accuracy: 19.12% 9 Validation loss: 339371.093750 Best loss: 16980.435547 Accuracy: 20.95% 10 Validation loss: 841728576.000000 Best loss: 16980.435547 Accuracy: 26.90% 11 Validation loss: 4837712896.000000 Best loss: 16980.435547 Accuracy: 22.01% 12 Validation loss: 20178567168.000000 Best loss: 16980.435547 Accuracy: 19.23% 13 Validation loss: 2399869184.000000 Best loss: 16980.435547 Accuracy: 19.23% 14 Validation loss: 1186248.375000 Best loss: 16980.435547 Accuracy: 18.73% 15 Validation loss: 575259.187500 Best loss: 16980.435547 Accuracy: 22.01% 16 Validation loss: 121365.570312 Best loss: 16980.435547 Accuracy: 29.05% 17 Validation loss: 116100.312500 Best loss: 16980.435547 Accuracy: 30.96% 18 Validation loss: 385177.875000 Best loss: 16980.435547 Accuracy: 22.17% 19 Validation loss: 726663.500000 Best loss: 16980.435547 Accuracy: 22.01% 20 Validation loss: 197889.015625 Best loss: 16980.435547 Accuracy: 19.35% 21 Validation loss: 602649216.000000 Best loss: 16980.435547 Accuracy: 19.16% 22 Validation loss: 17227335680.000000 Best loss: 16980.435547 Accuracy: 28.50% 23 Validation loss: 1475047296.000000 Best loss: 16980.435547 Accuracy: 19.08% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2, total= 2.4min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 2414026.500000 Best loss: 2414026.500000 Accuracy: 18.73% 1 Validation loss: 1236857.875000 Best loss: 1236857.875000 Accuracy: 18.73% 2 Validation loss: 237809.937500 Best loss: 237809.937500 Accuracy: 19.12% 3 Validation loss: 2893520.500000 Best loss: 237809.937500 Accuracy: 22.71% 4 Validation loss: 1129687.375000 Best loss: 237809.937500 Accuracy: 19.08% 5 Validation loss: 564067.437500 Best loss: 237809.937500 Accuracy: 26.90% 6 Validation loss: 1107041.875000 Best loss: 237809.937500 Accuracy: 25.92% 7 Validation loss: 632838.875000 Best loss: 237809.937500 Accuracy: 19.16% 8 Validation loss: 1054553.625000 Best loss: 237809.937500 Accuracy: 18.69% 9 Validation loss: 366879.343750 Best loss: 237809.937500 Accuracy: 23.30% 10 Validation loss: 173956.875000 Best loss: 173956.875000 Accuracy: 31.59% 11 Validation loss: 281153.781250 Best loss: 173956.875000 Accuracy: 27.21% 12 Validation loss: 1587033.250000 Best loss: 173956.875000 Accuracy: 20.84% 13 Validation loss: 443919.500000 Best loss: 173956.875000 Accuracy: 22.09% 14 Validation loss: 478460.656250 Best loss: 173956.875000 Accuracy: 22.01% 15 Validation loss: 319847.031250 Best loss: 173956.875000 Accuracy: 31.31% 16 Validation loss: 1978397.625000 Best loss: 173956.875000 Accuracy: 18.73% 17 Validation loss: 773913.062500 Best loss: 173956.875000 Accuracy: 22.01% 18 Validation loss: 635753.062500 Best loss: 173956.875000 Accuracy: 19.39% 19 Validation loss: 165957.609375 Best loss: 165957.609375 Accuracy: 38.43% 20 Validation loss: 415308288.000000 Best loss: 165957.609375 Accuracy: 18.57% 21 Validation loss: 2148251.500000 Best loss: 165957.609375 Accuracy: 22.09% 22 Validation loss: 5308701.000000 Best loss: 165957.609375 Accuracy: 19.08% 23 Validation loss: 1856707.000000 Best loss: 165957.609375 Accuracy: 20.91% 24 Validation loss: 1262596.500000 Best loss: 165957.609375 Accuracy: 22.01% 25 Validation loss: 321340.437500 Best loss: 165957.609375 Accuracy: 20.21% 26 Validation loss: 722468.687500 Best loss: 165957.609375 Accuracy: 19.12% 27 Validation loss: 743973.187500 Best loss: 165957.609375 Accuracy: 22.01% 28 Validation loss: 773088.812500 Best loss: 165957.609375 Accuracy: 18.49% 29 Validation loss: 409708.312500 Best loss: 165957.609375 Accuracy: 18.73% 30 Validation loss: 739213.187500 Best loss: 165957.609375 Accuracy: 18.73% 31 Validation loss: 61422260224.000000 Best loss: 165957.609375 Accuracy: 18.65% 32 Validation loss: 1182129.750000 Best loss: 165957.609375 Accuracy: 25.18% 33 Validation loss: 1423195.125000 Best loss: 165957.609375 Accuracy: 22.91% 34 Validation loss: 2753815.000000 Best loss: 165957.609375 Accuracy: 18.45% 35 Validation loss: 71695744.000000 Best loss: 165957.609375 Accuracy: 22.63% 36 Validation loss: 1652269.250000 Best loss: 165957.609375 Accuracy: 18.73% 37 Validation loss: 2067895.500000 Best loss: 165957.609375 Accuracy: 20.91% 38 Validation loss: 3399297.750000 Best loss: 165957.609375 Accuracy: 25.65% 39 Validation loss: 8453498.000000 Best loss: 165957.609375 Accuracy: 20.84% 40 Validation loss: 1934798.000000 Best loss: 165957.609375 Accuracy: 18.73% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2, total= 4.1min [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 800.930542 Best loss: 800.930542 Accuracy: 33.46% 1 Validation loss: 266.561829 Best loss: 266.561829 Accuracy: 38.12% 2 Validation loss: 249.727692 Best loss: 249.727692 Accuracy: 48.01% 3 Validation loss: 2316.195801 Best loss: 249.727692 Accuracy: 19.27% 4 Validation loss: 333608.625000 Best loss: 249.727692 Accuracy: 22.75% 5 Validation loss: 701701.062500 Best loss: 249.727692 Accuracy: 18.73% 6 Validation loss: 399610.250000 Best loss: 249.727692 Accuracy: 17.98% 7 Validation loss: 575455.437500 Best loss: 249.727692 Accuracy: 18.73% 8 Validation loss: 958473.125000 Best loss: 249.727692 Accuracy: 19.31% 9 Validation loss: 168685.265625 Best loss: 249.727692 Accuracy: 18.73% 10 Validation loss: 3499351.500000 Best loss: 249.727692 Accuracy: 20.95% 11 Validation loss: 205641.421875 Best loss: 249.727692 Accuracy: 19.90% 12 Validation loss: 241430.593750 Best loss: 249.727692 Accuracy: 18.73% 13 Validation loss: 208551.078125 Best loss: 249.727692 Accuracy: 20.25% 14 Validation loss: 708080.312500 Best loss: 249.727692 Accuracy: 37.88% 15 Validation loss: 4767011.500000 Best loss: 249.727692 Accuracy: 20.72% 16 Validation loss: 628464.312500 Best loss: 249.727692 Accuracy: 18.73% 17 Validation loss: 289204.593750 Best loss: 249.727692 Accuracy: 24.98% 18 Validation loss: 565001024.000000 Best loss: 249.727692 Accuracy: 22.71% 19 Validation loss: 38043066368.000000 Best loss: 249.727692 Accuracy: 19.23% 20 Validation loss: 86152003584.000000 Best loss: 249.727692 Accuracy: 22.01% 21 Validation loss: 131291799552.000000 Best loss: 249.727692 Accuracy: 23.10% 22 Validation loss: 161597145088.000000 Best loss: 249.727692 Accuracy: 20.09% 23 Validation loss: 167172915200.000000 Best loss: 249.727692 Accuracy: 22.01% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=100, batch_size=10, learning_rate=0.1, dropout_rate=0.2, total= 2.4min [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.245653 Best loss: 1.245653 Accuracy: 39.37% 1 Validation loss: 1.173521 Best loss: 1.173521 Accuracy: 40.19% 2 Validation loss: 1.147686 Best loss: 1.147686 Accuracy: 40.19% 3 Validation loss: 1.136884 Best loss: 1.136884 Accuracy: 40.66% 4 Validation loss: 1.269032 Best loss: 1.136884 Accuracy: 40.70% 5 Validation loss: 1.212851 Best loss: 1.136884 Accuracy: 40.38% 6 Validation loss: 1.127682 Best loss: 1.127682 Accuracy: 42.30% 7 Validation loss: 1.135265 Best loss: 1.127682 Accuracy: 40.38% 8 Validation loss: 1.108561 Best loss: 1.108561 Accuracy: 42.26% 9 Validation loss: 0.905645 Best loss: 0.905645 Accuracy: 60.83% 10 Validation loss: 0.783918 Best loss: 0.783918 Accuracy: 60.59% 11 Validation loss: 0.802063 Best loss: 0.783918 Accuracy: 60.71% 12 Validation loss: 0.877288 Best loss: 0.783918 Accuracy: 59.66% 13 Validation loss: 0.804724 Best loss: 0.783918 Accuracy: 60.32% 14 Validation loss: 0.843873 Best loss: 0.783918 Accuracy: 60.40% 15 Validation loss: 1.669260 Best loss: 0.783918 Accuracy: 20.91% 16 Validation loss: 1.626025 Best loss: 0.783918 Accuracy: 20.91% 17 Validation loss: 1.621203 Best loss: 0.783918 Accuracy: 19.27% 18 Validation loss: 1.690931 Best loss: 0.783918 Accuracy: 18.73% 19 Validation loss: 1.667967 Best loss: 0.783918 Accuracy: 22.01% 20 Validation loss: 1.659528 Best loss: 0.783918 Accuracy: 20.91% 21 Validation loss: 1.616188 Best loss: 0.783918 Accuracy: 22.01% 22 Validation loss: 1.666512 Best loss: 0.783918 Accuracy: 18.73% 23 Validation loss: 1.660243 Best loss: 0.783918 Accuracy: 19.27% 24 Validation loss: 1.717601 Best loss: 0.783918 Accuracy: 22.01% 25 Validation loss: 1.674895 Best loss: 0.783918 Accuracy: 22.01% 26 Validation loss: 1.634718 Best loss: 0.783918 Accuracy: 18.73% 27 Validation loss: 1.631264 Best loss: 0.783918 Accuracy: 18.73% 28 Validation loss: 1.625755 Best loss: 0.783918 Accuracy: 22.01% 29 Validation loss: 1.661761 Best loss: 0.783918 Accuracy: 19.27% 30 Validation loss: 1.677316 Best loss: 0.783918 Accuracy: 19.08% 31 Validation loss: 1.690778 Best loss: 0.783918 Accuracy: 22.01% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3, total= 5.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 0.932126 Best loss: 0.932126 Accuracy: 58.09% 1 Validation loss: 0.538145 Best loss: 0.538145 Accuracy: 74.04% 2 Validation loss: 0.412947 Best loss: 0.412947 Accuracy: 84.79% 3 Validation loss: 0.271034 Best loss: 0.271034 Accuracy: 93.94% 4 Validation loss: 0.175364 Best loss: 0.175364 Accuracy: 95.90% 5 Validation loss: 0.136967 Best loss: 0.136967 Accuracy: 97.38% 6 Validation loss: 0.131408 Best loss: 0.131408 Accuracy: 97.03% 7 Validation loss: 0.117977 Best loss: 0.117977 Accuracy: 97.73% 8 Validation loss: 0.116364 Best loss: 0.116364 Accuracy: 97.77% 9 Validation loss: 0.128745 Best loss: 0.116364 Accuracy: 97.58% 10 Validation loss: 0.115996 Best loss: 0.115996 Accuracy: 97.85% 11 Validation loss: 1.331812 Best loss: 0.115996 Accuracy: 48.12% 12 Validation loss: 1.620195 Best loss: 0.115996 Accuracy: 19.27% 13 Validation loss: 1.617375 Best loss: 0.115996 Accuracy: 20.91% 14 Validation loss: 1.625479 Best loss: 0.115996 Accuracy: 18.73% 15 Validation loss: 1.655699 Best loss: 0.115996 Accuracy: 19.08% 16 Validation loss: 1.642278 Best loss: 0.115996 Accuracy: 22.01% 17 Validation loss: 1.645422 Best loss: 0.115996 Accuracy: 19.08% 18 Validation loss: 1.673419 Best loss: 0.115996 Accuracy: 22.01% 19 Validation loss: 1.625313 Best loss: 0.115996 Accuracy: 19.08% 20 Validation loss: 1.613948 Best loss: 0.115996 Accuracy: 18.73% 21 Validation loss: 1.653298 Best loss: 0.115996 Accuracy: 22.01% 22 Validation loss: 1.639320 Best loss: 0.115996 Accuracy: 19.27% 23 Validation loss: 1.676577 Best loss: 0.115996 Accuracy: 22.01% 24 Validation loss: 1.653011 Best loss: 0.115996 Accuracy: 19.27% 25 Validation loss: 1.637113 Best loss: 0.115996 Accuracy: 19.27% 26 Validation loss: 1.633666 Best loss: 0.115996 Accuracy: 19.08% 27 Validation loss: 1.652607 Best loss: 0.115996 Accuracy: 18.73% 28 Validation loss: 1.664250 Best loss: 0.115996 Accuracy: 18.73% 29 Validation loss: 1.683340 Best loss: 0.115996 Accuracy: 18.73% 30 Validation loss: 1.618640 Best loss: 0.115996 Accuracy: 18.73% 31 Validation loss: 1.626807 Best loss: 0.115996 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3, total= 5.5s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3 0 Validation loss: 1.882889 Best loss: 1.882889 Accuracy: 18.73% 1 Validation loss: 1.255793 Best loss: 1.255793 Accuracy: 35.26% 2 Validation loss: 1.205383 Best loss: 1.205383 Accuracy: 37.41% 3 Validation loss: 1.175143 Best loss: 1.175143 Accuracy: 40.66% 4 Validation loss: 0.969054 Best loss: 0.969054 Accuracy: 55.28% 5 Validation loss: 0.718600 Best loss: 0.718600 Accuracy: 70.33% 6 Validation loss: 0.469740 Best loss: 0.469740 Accuracy: 87.53% 7 Validation loss: 0.260009 Best loss: 0.260009 Accuracy: 92.61% 8 Validation loss: 0.223210 Best loss: 0.223210 Accuracy: 94.21% 9 Validation loss: 0.259290 Best loss: 0.223210 Accuracy: 94.33% 10 Validation loss: 0.226821 Best loss: 0.223210 Accuracy: 96.01% 11 Validation loss: 0.178013 Best loss: 0.178013 Accuracy: 96.21% 12 Validation loss: 0.180874 Best loss: 0.178013 Accuracy: 95.86% 13 Validation loss: 0.152422 Best loss: 0.152422 Accuracy: 96.25% 14 Validation loss: 0.176338 Best loss: 0.152422 Accuracy: 96.64% 15 Validation loss: 0.158895 Best loss: 0.152422 Accuracy: 96.33% 16 Validation loss: 0.130984 Best loss: 0.130984 Accuracy: 96.83% 17 Validation loss: 0.139437 Best loss: 0.130984 Accuracy: 96.91% 18 Validation loss: 0.169439 Best loss: 0.130984 Accuracy: 96.44% 19 Validation loss: 0.147996 Best loss: 0.130984 Accuracy: 97.11% 20 Validation loss: 0.128239 Best loss: 0.128239 Accuracy: 97.34% 21 Validation loss: 0.138903 Best loss: 0.128239 Accuracy: 97.26% 22 Validation loss: 0.158770 Best loss: 0.128239 Accuracy: 97.26% 23 Validation loss: 0.136611 Best loss: 0.128239 Accuracy: 97.30% 24 Validation loss: 0.149716 Best loss: 0.128239 Accuracy: 96.99% 25 Validation loss: 0.141083 Best loss: 0.128239 Accuracy: 97.46% 26 Validation loss: 0.148741 Best loss: 0.128239 Accuracy: 96.64% 27 Validation loss: 1.817967 Best loss: 0.128239 Accuracy: 19.08% 28 Validation loss: 1.713573 Best loss: 0.128239 Accuracy: 19.08% 29 Validation loss: 1.614016 Best loss: 0.128239 Accuracy: 22.01% 30 Validation loss: 1.650740 Best loss: 0.128239 Accuracy: 19.08% 31 Validation loss: 1.625108 Best loss: 0.128239 Accuracy: 18.73% 32 Validation loss: 1.644863 Best loss: 0.128239 Accuracy: 18.73% 33 Validation loss: 1.610969 Best loss: 0.128239 Accuracy: 22.01% 34 Validation loss: 1.616402 Best loss: 0.128239 Accuracy: 22.01% 35 Validation loss: 1.623172 Best loss: 0.128239 Accuracy: 22.01% 36 Validation loss: 1.609914 Best loss: 0.128239 Accuracy: 20.91% 37 Validation loss: 1.619347 Best loss: 0.128239 Accuracy: 22.01% 38 Validation loss: 1.633823 Best loss: 0.128239 Accuracy: 18.73% 39 Validation loss: 1.717980 Best loss: 0.128239 Accuracy: 19.27% 40 Validation loss: 1.663784 Best loss: 0.128239 Accuracy: 22.01% 41 Validation loss: 1.616033 Best loss: 0.128239 Accuracy: 18.73% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=500, learning_rate=0.05, dropout_rate=0.3, total= 7.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 663.237976 Best loss: 663.237976 Accuracy: 38.62% 1 Validation loss: 300.625061 Best loss: 300.625061 Accuracy: 27.40% 2 Validation loss: 175.890549 Best loss: 175.890549 Accuracy: 34.68% 3 Validation loss: 78.542931 Best loss: 78.542931 Accuracy: 34.87% 4 Validation loss: 110.082817 Best loss: 78.542931 Accuracy: 52.42% 5 Validation loss: 22.040718 Best loss: 22.040718 Accuracy: 53.60% 6 Validation loss: 119.163460 Best loss: 22.040718 Accuracy: 38.15% 7 Validation loss: 178.322937 Best loss: 22.040718 Accuracy: 39.52% 8 Validation loss: 81848.273438 Best loss: 22.040718 Accuracy: 59.38% 9 Validation loss: 49524.769531 Best loss: 22.040718 Accuracy: 54.61% 10 Validation loss: 7634.599121 Best loss: 22.040718 Accuracy: 57.43% 11 Validation loss: 4463.614746 Best loss: 22.040718 Accuracy: 70.21% 12 Validation loss: 13680.067383 Best loss: 22.040718 Accuracy: 67.51% 13 Validation loss: 5657.843750 Best loss: 22.040718 Accuracy: 74.43% 14 Validation loss: 3595.489502 Best loss: 22.040718 Accuracy: 73.77% 15 Validation loss: 3083.383789 Best loss: 22.040718 Accuracy: 87.57% 16 Validation loss: 2398.089600 Best loss: 22.040718 Accuracy: 69.31% 17 Validation loss: 4752.446777 Best loss: 22.040718 Accuracy: 65.64% 18 Validation loss: 2043.011963 Best loss: 22.040718 Accuracy: 70.99% 19 Validation loss: 978.777954 Best loss: 22.040718 Accuracy: 89.72% 20 Validation loss: 2499.595947 Best loss: 22.040718 Accuracy: 75.33% 21 Validation loss: 2104.687988 Best loss: 22.040718 Accuracy: 69.82% 22 Validation loss: 376.289764 Best loss: 22.040718 Accuracy: 84.21% 23 Validation loss: 6541.246094 Best loss: 22.040718 Accuracy: 67.98% 24 Validation loss: 356506.968750 Best loss: 22.040718 Accuracy: 65.01% 25 Validation loss: 120709.507812 Best loss: 22.040718 Accuracy: 68.92% 26 Validation loss: 98766.195312 Best loss: 22.040718 Accuracy: 68.02% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 34.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 185.268051 Best loss: 185.268051 Accuracy: 32.64% 1 Validation loss: 9838.005859 Best loss: 185.268051 Accuracy: 20.88% 2 Validation loss: 2488.015381 Best loss: 185.268051 Accuracy: 33.89% 3 Validation loss: 1250.239502 Best loss: 185.268051 Accuracy: 30.57% 4 Validation loss: 1658.424561 Best loss: 185.268051 Accuracy: 24.47% 5 Validation loss: 1112.195190 Best loss: 185.268051 Accuracy: 39.01% 6 Validation loss: 376.689331 Best loss: 185.268051 Accuracy: 39.33% 7 Validation loss: 526.615540 Best loss: 185.268051 Accuracy: 35.03% 8 Validation loss: 450.389130 Best loss: 185.268051 Accuracy: 22.75% 9 Validation loss: 311.739502 Best loss: 185.268051 Accuracy: 36.79% 10 Validation loss: 119.009712 Best loss: 119.009712 Accuracy: 63.37% 11 Validation loss: 328.949493 Best loss: 119.009712 Accuracy: 46.09% 12 Validation loss: 704.744934 Best loss: 119.009712 Accuracy: 37.49% 13 Validation loss: 941.809814 Best loss: 119.009712 Accuracy: 33.23% 14 Validation loss: 53.647007 Best loss: 53.647007 Accuracy: 55.36% 15 Validation loss: 40.985970 Best loss: 40.985970 Accuracy: 65.17% 16 Validation loss: 65.935814 Best loss: 40.985970 Accuracy: 57.35% 17 Validation loss: 49.864262 Best loss: 40.985970 Accuracy: 56.14% 18 Validation loss: 30.342937 Best loss: 30.342937 Accuracy: 65.29% 19 Validation loss: 3781.665039 Best loss: 30.342937 Accuracy: 29.32% 20 Validation loss: 163169.500000 Best loss: 30.342937 Accuracy: 59.93% 21 Validation loss: 60396.640625 Best loss: 30.342937 Accuracy: 40.89% 22 Validation loss: 82301.382812 Best loss: 30.342937 Accuracy: 58.09% 23 Validation loss: 86348.062500 Best loss: 30.342937 Accuracy: 60.95% 24 Validation loss: 54112.460938 Best loss: 30.342937 Accuracy: 56.84% 25 Validation loss: 12001.329102 Best loss: 30.342937 Accuracy: 63.02% 26 Validation loss: 58335.500000 Best loss: 30.342937 Accuracy: 54.96% 27 Validation loss: 17026.029297 Best loss: 30.342937 Accuracy: 63.64% 28 Validation loss: 8302.532227 Best loss: 30.342937 Accuracy: 68.69% 29 Validation loss: 507788.406250 Best loss: 30.342937 Accuracy: 53.01% 30 Validation loss: 322222.000000 Best loss: 30.342937 Accuracy: 55.08% 31 Validation loss: 31727.369141 Best loss: 30.342937 Accuracy: 55.43% 32 Validation loss: 23395.980469 Best loss: 30.342937 Accuracy: 56.02% 33 Validation loss: 21337.291016 Best loss: 30.342937 Accuracy: 56.14% 34 Validation loss: 10194.720703 Best loss: 30.342937 Accuracy: 59.77% 35 Validation loss: 12318.142578 Best loss: 30.342937 Accuracy: 59.58% 36 Validation loss: 11875.165039 Best loss: 30.342937 Accuracy: 56.18% 37 Validation loss: 6793.938965 Best loss: 30.342937 Accuracy: 70.21% 38 Validation loss: 6444.444336 Best loss: 30.342937 Accuracy: 59.62% 39 Validation loss: 4034.799561 Best loss: 30.342937 Accuracy: 57.35% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 49.5s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2 0 Validation loss: 1249.104248 Best loss: 1249.104248 Accuracy: 42.30% 1 Validation loss: 569.829712 Best loss: 569.829712 Accuracy: 59.81% 2 Validation loss: 236.161057 Best loss: 236.161057 Accuracy: 70.48% 3 Validation loss: 383417.281250 Best loss: 236.161057 Accuracy: 19.35% 4 Validation loss: 10989.544922 Best loss: 236.161057 Accuracy: 35.69% 5 Validation loss: 525.165100 Best loss: 236.161057 Accuracy: 56.10% 6 Validation loss: 4756.269531 Best loss: 236.161057 Accuracy: 31.67% 7 Validation loss: 2765.040771 Best loss: 236.161057 Accuracy: 35.26% 8 Validation loss: 6135.742676 Best loss: 236.161057 Accuracy: 39.05% 9 Validation loss: 990.258606 Best loss: 236.161057 Accuracy: 59.50% 10 Validation loss: 1621.620483 Best loss: 236.161057 Accuracy: 54.07% 11 Validation loss: 1329.005127 Best loss: 236.161057 Accuracy: 54.18% 12 Validation loss: 716.494446 Best loss: 236.161057 Accuracy: 55.67% 13 Validation loss: 138.117081 Best loss: 138.117081 Accuracy: 70.45% 14 Validation loss: 786.825867 Best loss: 138.117081 Accuracy: 54.18% 15 Validation loss: 586.288025 Best loss: 138.117081 Accuracy: 64.89% 16 Validation loss: 8702227.000000 Best loss: 138.117081 Accuracy: 31.51% 17 Validation loss: 174062.859375 Best loss: 138.117081 Accuracy: 72.28% 18 Validation loss: 97950.226562 Best loss: 138.117081 Accuracy: 75.65% 19 Validation loss: 192224.109375 Best loss: 138.117081 Accuracy: 68.45% 20 Validation loss: 361574.687500 Best loss: 138.117081 Accuracy: 54.61% 21 Validation loss: 10622.606445 Best loss: 138.117081 Accuracy: 87.65% 22 Validation loss: 33367.707031 Best loss: 138.117081 Accuracy: 77.01% 23 Validation loss: 58405.855469 Best loss: 138.117081 Accuracy: 72.71% 24 Validation loss: 445096.531250 Best loss: 138.117081 Accuracy: 79.01% 25 Validation loss: 210421.140625 Best loss: 138.117081 Accuracy: 81.20% 26 Validation loss: 126741.289062 Best loss: 138.117081 Accuracy: 69.82% 27 Validation loss: 90139.382812 Best loss: 138.117081 Accuracy: 81.51% 28 Validation loss: 34456.929688 Best loss: 138.117081 Accuracy: 91.20% 29 Validation loss: 53463.253906 Best loss: 138.117081 Accuracy: 76.23% 30 Validation loss: 75917.648438 Best loss: 138.117081 Accuracy: 82.92% 31 Validation loss: 53193.218750 Best loss: 138.117081 Accuracy: 84.56% 32 Validation loss: 36892.804688 Best loss: 138.117081 Accuracy: 80.88% 33 Validation loss: 42449.519531 Best loss: 138.117081 Accuracy: 91.24% 34 Validation loss: 60302.410156 Best loss: 138.117081 Accuracy: 71.81% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=70, batch_size=50, learning_rate=0.1, dropout_rate=0.2, total= 44.6s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.552150 Best loss: 0.552150 Accuracy: 86.63% 1 Validation loss: 0.214820 Best loss: 0.214820 Accuracy: 94.88% 2 Validation loss: 0.169988 Best loss: 0.169988 Accuracy: 95.47% 3 Validation loss: 0.157689 Best loss: 0.157689 Accuracy: 95.74% 4 Validation loss: 0.151566 Best loss: 0.151566 Accuracy: 95.93% 5 Validation loss: 0.157243 Best loss: 0.151566 Accuracy: 95.90% 6 Validation loss: 0.154677 Best loss: 0.151566 Accuracy: 95.78% 7 Validation loss: 0.147131 Best loss: 0.147131 Accuracy: 96.09% 8 Validation loss: 0.147868 Best loss: 0.147131 Accuracy: 96.25% 9 Validation loss: 0.141991 Best loss: 0.141991 Accuracy: 96.44% 10 Validation loss: 0.145446 Best loss: 0.141991 Accuracy: 96.21% 11 Validation loss: 0.138777 Best loss: 0.138777 Accuracy: 96.52% 12 Validation loss: 0.157380 Best loss: 0.138777 Accuracy: 96.17% 13 Validation loss: 0.162192 Best loss: 0.138777 Accuracy: 96.40% 14 Validation loss: 0.149090 Best loss: 0.138777 Accuracy: 96.48% 15 Validation loss: 0.149395 Best loss: 0.138777 Accuracy: 96.25% 16 Validation loss: 0.145762 Best loss: 0.138777 Accuracy: 96.48% 17 Validation loss: 0.186282 Best loss: 0.138777 Accuracy: 96.17% 18 Validation loss: 0.156899 Best loss: 0.138777 Accuracy: 96.21% 19 Validation loss: 0.138806 Best loss: 0.138777 Accuracy: 96.17% 20 Validation loss: 0.139896 Best loss: 0.138777 Accuracy: 96.36% 21 Validation loss: 0.147075 Best loss: 0.138777 Accuracy: 96.60% 22 Validation loss: 0.143392 Best loss: 0.138777 Accuracy: 96.40% 23 Validation loss: 0.149699 Best loss: 0.138777 Accuracy: 96.40% 24 Validation loss: 0.156381 Best loss: 0.138777 Accuracy: 96.52% 25 Validation loss: 0.147247 Best loss: 0.138777 Accuracy: 96.36% 26 Validation loss: 0.150442 Best loss: 0.138777 Accuracy: 96.44% 27 Validation loss: 0.158240 Best loss: 0.138777 Accuracy: 96.68% 28 Validation loss: 0.139907 Best loss: 0.138777 Accuracy: 96.64% 29 Validation loss: 0.155313 Best loss: 0.138777 Accuracy: 96.44% 30 Validation loss: 0.141904 Best loss: 0.138777 Accuracy: 96.60% 31 Validation loss: 0.146380 Best loss: 0.138777 Accuracy: 96.68% 32 Validation loss: 0.143513 Best loss: 0.138777 Accuracy: 96.60% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6, total= 5.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.496251 Best loss: 0.496251 Accuracy: 81.39% 1 Validation loss: 0.383576 Best loss: 0.383576 Accuracy: 90.77% 2 Validation loss: 0.268740 Best loss: 0.268740 Accuracy: 94.45% 3 Validation loss: 0.184715 Best loss: 0.184715 Accuracy: 95.47% 4 Validation loss: 0.178195 Best loss: 0.178195 Accuracy: 95.62% 5 Validation loss: 0.159722 Best loss: 0.159722 Accuracy: 95.54% 6 Validation loss: 0.144051 Best loss: 0.144051 Accuracy: 96.17% 7 Validation loss: 0.156280 Best loss: 0.144051 Accuracy: 95.90% 8 Validation loss: 0.154129 Best loss: 0.144051 Accuracy: 96.13% 9 Validation loss: 0.163483 Best loss: 0.144051 Accuracy: 96.17% 10 Validation loss: 0.149285 Best loss: 0.144051 Accuracy: 96.17% 11 Validation loss: 0.147706 Best loss: 0.144051 Accuracy: 96.17% 12 Validation loss: 0.153318 Best loss: 0.144051 Accuracy: 96.29% 13 Validation loss: 0.154448 Best loss: 0.144051 Accuracy: 96.44% 14 Validation loss: 0.144067 Best loss: 0.144051 Accuracy: 96.44% 15 Validation loss: 0.140531 Best loss: 0.140531 Accuracy: 96.29% 16 Validation loss: 0.170954 Best loss: 0.140531 Accuracy: 96.44% 17 Validation loss: 0.135960 Best loss: 0.135960 Accuracy: 96.48% 18 Validation loss: 0.144351 Best loss: 0.135960 Accuracy: 96.21% 19 Validation loss: 0.138556 Best loss: 0.135960 Accuracy: 96.40% 20 Validation loss: 0.152143 Best loss: 0.135960 Accuracy: 96.52% 21 Validation loss: 0.136336 Best loss: 0.135960 Accuracy: 96.60% 22 Validation loss: 0.125432 Best loss: 0.125432 Accuracy: 96.91% 23 Validation loss: 0.150172 Best loss: 0.125432 Accuracy: 96.48% 24 Validation loss: 0.135279 Best loss: 0.125432 Accuracy: 96.60% 25 Validation loss: 0.138970 Best loss: 0.125432 Accuracy: 96.44% 26 Validation loss: 0.140168 Best loss: 0.125432 Accuracy: 96.48% 27 Validation loss: 0.147047 Best loss: 0.125432 Accuracy: 96.48% 28 Validation loss: 0.143107 Best loss: 0.125432 Accuracy: 96.36% 29 Validation loss: 0.143937 Best loss: 0.125432 Accuracy: 96.33% 30 Validation loss: 0.140026 Best loss: 0.125432 Accuracy: 96.68% 31 Validation loss: 0.142670 Best loss: 0.125432 Accuracy: 96.44% 32 Validation loss: 0.127136 Best loss: 0.125432 Accuracy: 96.91% 33 Validation loss: 0.130415 Best loss: 0.125432 Accuracy: 96.83% 34 Validation loss: 0.132018 Best loss: 0.125432 Accuracy: 96.72% 35 Validation loss: 0.139102 Best loss: 0.125432 Accuracy: 96.52% 36 Validation loss: 0.129498 Best loss: 0.125432 Accuracy: 96.83% 37 Validation loss: 0.134962 Best loss: 0.125432 Accuracy: 96.72% 38 Validation loss: 0.134248 Best loss: 0.125432 Accuracy: 96.60% 39 Validation loss: 0.141466 Best loss: 0.125432 Accuracy: 96.60% 40 Validation loss: 0.141918 Best loss: 0.125432 Accuracy: 96.72% 41 Validation loss: 0.145099 Best loss: 0.125432 Accuracy: 96.29% 42 Validation loss: 0.148191 Best loss: 0.125432 Accuracy: 96.36% 43 Validation loss: 0.144670 Best loss: 0.125432 Accuracy: 96.60% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6, total= 7.8s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6 0 Validation loss: 0.411620 Best loss: 0.411620 Accuracy: 89.09% 1 Validation loss: 0.308153 Best loss: 0.308153 Accuracy: 93.63% 2 Validation loss: 0.223227 Best loss: 0.223227 Accuracy: 94.57% 3 Validation loss: 0.165893 Best loss: 0.165893 Accuracy: 95.54% 4 Validation loss: 0.163090 Best loss: 0.163090 Accuracy: 95.74% 5 Validation loss: 0.160990 Best loss: 0.160990 Accuracy: 96.01% 6 Validation loss: 0.168185 Best loss: 0.160990 Accuracy: 96.09% 7 Validation loss: 0.150497 Best loss: 0.150497 Accuracy: 96.36% 8 Validation loss: 0.170031 Best loss: 0.150497 Accuracy: 96.13% 9 Validation loss: 0.159370 Best loss: 0.150497 Accuracy: 96.40% 10 Validation loss: 0.149324 Best loss: 0.149324 Accuracy: 96.33% 11 Validation loss: 0.154442 Best loss: 0.149324 Accuracy: 96.33% 12 Validation loss: 0.132207 Best loss: 0.132207 Accuracy: 96.72% 13 Validation loss: 0.153146 Best loss: 0.132207 Accuracy: 96.40% 14 Validation loss: 0.135864 Best loss: 0.132207 Accuracy: 96.52% 15 Validation loss: 0.146414 Best loss: 0.132207 Accuracy: 96.40% 16 Validation loss: 0.146572 Best loss: 0.132207 Accuracy: 96.64% 17 Validation loss: 0.141913 Best loss: 0.132207 Accuracy: 96.76% 18 Validation loss: 0.142172 Best loss: 0.132207 Accuracy: 96.60% 19 Validation loss: 0.147912 Best loss: 0.132207 Accuracy: 96.44% 20 Validation loss: 0.146406 Best loss: 0.132207 Accuracy: 96.87% 21 Validation loss: 0.127016 Best loss: 0.127016 Accuracy: 96.68% 22 Validation loss: 0.135591 Best loss: 0.127016 Accuracy: 96.83% 23 Validation loss: 0.143149 Best loss: 0.127016 Accuracy: 96.36% 24 Validation loss: 0.144359 Best loss: 0.127016 Accuracy: 96.48% 25 Validation loss: 0.148938 Best loss: 0.127016 Accuracy: 96.40% 26 Validation loss: 0.141820 Best loss: 0.127016 Accuracy: 96.48% 27 Validation loss: 0.150963 Best loss: 0.127016 Accuracy: 96.40% 28 Validation loss: 0.162103 Best loss: 0.127016 Accuracy: 96.44% 29 Validation loss: 0.135434 Best loss: 0.127016 Accuracy: 96.68% 30 Validation loss: 0.128463 Best loss: 0.127016 Accuracy: 96.99% 31 Validation loss: 0.127752 Best loss: 0.127016 Accuracy: 96.99% 32 Validation loss: 0.137685 Best loss: 0.127016 Accuracy: 96.64% 33 Validation loss: 0.133367 Best loss: 0.127016 Accuracy: 96.68% 34 Validation loss: 0.157302 Best loss: 0.127016 Accuracy: 96.56% 35 Validation loss: 0.140129 Best loss: 0.127016 Accuracy: 96.79% 36 Validation loss: 0.137282 Best loss: 0.127016 Accuracy: 96.68% 37 Validation loss: 0.143128 Best loss: 0.127016 Accuracy: 96.40% 38 Validation loss: 0.138249 Best loss: 0.127016 Accuracy: 96.68% 39 Validation loss: 0.146075 Best loss: 0.127016 Accuracy: 96.76% 40 Validation loss: 0.127716 Best loss: 0.127016 Accuracy: 96.76% 41 Validation loss: 0.137535 Best loss: 0.127016 Accuracy: 96.64% 42 Validation loss: 0.127632 Best loss: 0.127016 Accuracy: 96.79% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=30, batch_size=500, learning_rate=0.01, dropout_rate=0.6, total= 7.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.147766 Best loss: 0.147766 Accuracy: 96.01% 1 Validation loss: 0.106580 Best loss: 0.106580 Accuracy: 97.19% 2 Validation loss: 0.086305 Best loss: 0.086305 Accuracy: 97.77% 3 Validation loss: 0.085875 Best loss: 0.085875 Accuracy: 98.01% 4 Validation loss: 0.074927 Best loss: 0.074927 Accuracy: 97.93% 5 Validation loss: 0.089905 Best loss: 0.074927 Accuracy: 97.65% 6 Validation loss: 0.073685 Best loss: 0.073685 Accuracy: 98.16% 7 Validation loss: 0.077966 Best loss: 0.073685 Accuracy: 98.01% 8 Validation loss: 0.067326 Best loss: 0.067326 Accuracy: 98.28% 9 Validation loss: 0.084536 Best loss: 0.067326 Accuracy: 98.05% 10 Validation loss: 0.083942 Best loss: 0.067326 Accuracy: 98.01% 11 Validation loss: 0.076103 Best loss: 0.067326 Accuracy: 98.08% 12 Validation loss: 0.076466 Best loss: 0.067326 Accuracy: 98.01% 13 Validation loss: 0.077225 Best loss: 0.067326 Accuracy: 98.32% 14 Validation loss: 0.109172 Best loss: 0.067326 Accuracy: 97.93% 15 Validation loss: 0.102606 Best loss: 0.067326 Accuracy: 97.42% 16 Validation loss: 0.089911 Best loss: 0.067326 Accuracy: 98.01% 17 Validation loss: 0.088629 Best loss: 0.067326 Accuracy: 98.16% 18 Validation loss: 0.123294 Best loss: 0.067326 Accuracy: 97.89% 19 Validation loss: 0.111370 Best loss: 0.067326 Accuracy: 97.34% 20 Validation loss: 0.110887 Best loss: 0.067326 Accuracy: 97.26% 21 Validation loss: 0.122138 Best loss: 0.067326 Accuracy: 96.52% 22 Validation loss: 0.109607 Best loss: 0.067326 Accuracy: 97.30% 23 Validation loss: 0.104470 Best loss: 0.067326 Accuracy: 97.22% 24 Validation loss: 0.224250 Best loss: 0.067326 Accuracy: 95.19% 25 Validation loss: 0.183129 Best loss: 0.067326 Accuracy: 95.74% 26 Validation loss: 0.138970 Best loss: 0.067326 Accuracy: 96.52% 27 Validation loss: 0.346106 Best loss: 0.067326 Accuracy: 89.01% 28 Validation loss: 0.175546 Best loss: 0.067326 Accuracy: 94.57% 29 Validation loss: 0.143110 Best loss: 0.067326 Accuracy: 95.97% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3, total= 6.0s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.122858 Best loss: 0.122858 Accuracy: 96.60% 1 Validation loss: 0.091870 Best loss: 0.091870 Accuracy: 97.54% 2 Validation loss: 0.075684 Best loss: 0.075684 Accuracy: 98.01% 3 Validation loss: 0.081823 Best loss: 0.075684 Accuracy: 97.65% 4 Validation loss: 0.080213 Best loss: 0.075684 Accuracy: 97.85% 5 Validation loss: 0.076183 Best loss: 0.075684 Accuracy: 97.85% 6 Validation loss: 0.066903 Best loss: 0.066903 Accuracy: 98.12% 7 Validation loss: 0.067005 Best loss: 0.066903 Accuracy: 98.12% 8 Validation loss: 0.071191 Best loss: 0.066903 Accuracy: 98.28% 9 Validation loss: 0.073940 Best loss: 0.066903 Accuracy: 98.12% 10 Validation loss: 0.074013 Best loss: 0.066903 Accuracy: 98.05% 11 Validation loss: 0.078948 Best loss: 0.066903 Accuracy: 98.12% 12 Validation loss: 0.088992 Best loss: 0.066903 Accuracy: 97.77% 13 Validation loss: 0.073776 Best loss: 0.066903 Accuracy: 97.77% 14 Validation loss: 0.090169 Best loss: 0.066903 Accuracy: 97.58% 15 Validation loss: 0.134303 Best loss: 0.066903 Accuracy: 96.79% 16 Validation loss: 0.156693 Best loss: 0.066903 Accuracy: 94.68% 17 Validation loss: 0.113509 Best loss: 0.066903 Accuracy: 97.42% 18 Validation loss: 0.107157 Best loss: 0.066903 Accuracy: 96.87% 19 Validation loss: 0.089374 Best loss: 0.066903 Accuracy: 97.62% 20 Validation loss: 0.079296 Best loss: 0.066903 Accuracy: 97.62% 21 Validation loss: 0.077442 Best loss: 0.066903 Accuracy: 97.69% 22 Validation loss: 0.070250 Best loss: 0.066903 Accuracy: 97.85% 23 Validation loss: 0.148700 Best loss: 0.066903 Accuracy: 96.21% 24 Validation loss: 0.112214 Best loss: 0.066903 Accuracy: 97.19% 25 Validation loss: 0.085684 Best loss: 0.066903 Accuracy: 97.85% 26 Validation loss: 0.108473 Best loss: 0.066903 Accuracy: 97.50% 27 Validation loss: 0.091605 Best loss: 0.066903 Accuracy: 97.62% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3, total= 5.8s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3 0 Validation loss: 0.132954 Best loss: 0.132954 Accuracy: 96.40% 1 Validation loss: 0.111452 Best loss: 0.111452 Accuracy: 96.99% 2 Validation loss: 0.094436 Best loss: 0.094436 Accuracy: 97.54% 3 Validation loss: 0.093583 Best loss: 0.093583 Accuracy: 97.77% 4 Validation loss: 0.085847 Best loss: 0.085847 Accuracy: 97.93% 5 Validation loss: 0.088511 Best loss: 0.085847 Accuracy: 97.85% 6 Validation loss: 0.075020 Best loss: 0.075020 Accuracy: 98.24% 7 Validation loss: 0.068253 Best loss: 0.068253 Accuracy: 98.16% 8 Validation loss: 0.074498 Best loss: 0.068253 Accuracy: 98.28% 9 Validation loss: 0.079685 Best loss: 0.068253 Accuracy: 98.32% 10 Validation loss: 0.067688 Best loss: 0.067688 Accuracy: 98.08% 11 Validation loss: 0.069342 Best loss: 0.067688 Accuracy: 98.40% 12 Validation loss: 0.055763 Best loss: 0.055763 Accuracy: 98.59% 13 Validation loss: 0.064105 Best loss: 0.055763 Accuracy: 98.44% 14 Validation loss: 0.065981 Best loss: 0.055763 Accuracy: 98.44% 15 Validation loss: 0.075998 Best loss: 0.055763 Accuracy: 98.20% 16 Validation loss: 0.079497 Best loss: 0.055763 Accuracy: 97.89% 17 Validation loss: 0.092217 Best loss: 0.055763 Accuracy: 97.62% 18 Validation loss: 0.108108 Best loss: 0.055763 Accuracy: 96.76% 19 Validation loss: 0.127938 Best loss: 0.055763 Accuracy: 96.13% 20 Validation loss: 0.133652 Best loss: 0.055763 Accuracy: 96.40% 21 Validation loss: 0.135753 Best loss: 0.055763 Accuracy: 97.46% 22 Validation loss: 0.135523 Best loss: 0.055763 Accuracy: 95.86% 23 Validation loss: 0.140442 Best loss: 0.055763 Accuracy: 96.64% 24 Validation loss: 0.164001 Best loss: 0.055763 Accuracy: 96.05% 25 Validation loss: 0.109513 Best loss: 0.055763 Accuracy: 97.15% 26 Validation loss: 0.100193 Best loss: 0.055763 Accuracy: 97.54% 27 Validation loss: 0.818256 Best loss: 0.055763 Accuracy: 94.25% 28 Validation loss: 0.341496 Best loss: 0.055763 Accuracy: 91.09% 29 Validation loss: 0.234354 Best loss: 0.055763 Accuracy: 93.98% 30 Validation loss: 0.213720 Best loss: 0.055763 Accuracy: 94.33% 31 Validation loss: 0.207097 Best loss: 0.055763 Accuracy: 95.62% 32 Validation loss: 0.214993 Best loss: 0.055763 Accuracy: 93.98% 33 Validation loss: 0.222104 Best loss: 0.055763 Accuracy: 91.48% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=90, batch_size=500, learning_rate=0.02, dropout_rate=0.3, total= 6.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.100512 Best loss: 0.100512 Accuracy: 97.42% 1 Validation loss: 0.078490 Best loss: 0.078490 Accuracy: 98.32% 2 Validation loss: 0.085431 Best loss: 0.078490 Accuracy: 97.54% 3 Validation loss: 0.085063 Best loss: 0.078490 Accuracy: 97.62% 4 Validation loss: 0.079306 Best loss: 0.078490 Accuracy: 98.12% 5 Validation loss: 0.095474 Best loss: 0.078490 Accuracy: 98.20% 6 Validation loss: 0.059441 Best loss: 0.059441 Accuracy: 98.36% 7 Validation loss: 0.079952 Best loss: 0.059441 Accuracy: 98.01% 8 Validation loss: 0.071697 Best loss: 0.059441 Accuracy: 98.24% 9 Validation loss: 0.059378 Best loss: 0.059378 Accuracy: 98.48% 10 Validation loss: 0.075235 Best loss: 0.059378 Accuracy: 98.24% 11 Validation loss: 0.074882 Best loss: 0.059378 Accuracy: 98.44% 12 Validation loss: 0.065513 Best loss: 0.059378 Accuracy: 98.28% 13 Validation loss: 0.076620 Best loss: 0.059378 Accuracy: 98.05% 14 Validation loss: 0.066921 Best loss: 0.059378 Accuracy: 98.24% 15 Validation loss: 0.116741 Best loss: 0.059378 Accuracy: 97.69% 16 Validation loss: 0.151334 Best loss: 0.059378 Accuracy: 97.19% 17 Validation loss: 0.193541 Best loss: 0.059378 Accuracy: 97.03% 18 Validation loss: 0.075451 Best loss: 0.059378 Accuracy: 98.01% 19 Validation loss: 0.059616 Best loss: 0.059378 Accuracy: 98.24% 20 Validation loss: 0.064906 Best loss: 0.059378 Accuracy: 98.44% 21 Validation loss: 0.061472 Best loss: 0.059378 Accuracy: 98.36% 22 Validation loss: 0.064161 Best loss: 0.059378 Accuracy: 98.55% 23 Validation loss: 0.058115 Best loss: 0.058115 Accuracy: 98.51% 24 Validation loss: 0.051032 Best loss: 0.051032 Accuracy: 98.79% 25 Validation loss: 0.050230 Best loss: 0.050230 Accuracy: 98.63% 26 Validation loss: 0.050093 Best loss: 0.050093 Accuracy: 98.67% 27 Validation loss: 0.051442 Best loss: 0.050093 Accuracy: 98.87% 28 Validation loss: 0.048122 Best loss: 0.048122 Accuracy: 98.63% 29 Validation loss: 0.053781 Best loss: 0.048122 Accuracy: 98.63% 30 Validation loss: 0.058376 Best loss: 0.048122 Accuracy: 98.20% 31 Validation loss: 0.052468 Best loss: 0.048122 Accuracy: 98.67% 32 Validation loss: 0.050390 Best loss: 0.048122 Accuracy: 98.83% 33 Validation loss: 0.088953 Best loss: 0.048122 Accuracy: 97.58% 34 Validation loss: 0.073336 Best loss: 0.048122 Accuracy: 98.59% 35 Validation loss: 0.065937 Best loss: 0.048122 Accuracy: 98.63% 36 Validation loss: 0.062092 Best loss: 0.048122 Accuracy: 98.59% 37 Validation loss: 0.082520 Best loss: 0.048122 Accuracy: 98.28% 38 Validation loss: 0.078820 Best loss: 0.048122 Accuracy: 98.12% 39 Validation loss: 0.065200 Best loss: 0.048122 Accuracy: 98.28% 40 Validation loss: 0.098976 Best loss: 0.048122 Accuracy: 98.32% 41 Validation loss: 0.080310 Best loss: 0.048122 Accuracy: 98.79% 42 Validation loss: 0.062366 Best loss: 0.048122 Accuracy: 98.67% 43 Validation loss: 0.077084 Best loss: 0.048122 Accuracy: 98.55% 44 Validation loss: 0.065308 Best loss: 0.048122 Accuracy: 98.59% 45 Validation loss: 0.086916 Best loss: 0.048122 Accuracy: 98.67% 46 Validation loss: 0.082961 Best loss: 0.048122 Accuracy: 98.40% 47 Validation loss: 0.070788 Best loss: 0.048122 Accuracy: 98.51% 48 Validation loss: 0.062719 Best loss: 0.048122 Accuracy: 98.63% 49 Validation loss: 0.054789 Best loss: 0.048122 Accuracy: 98.75% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 34.2s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.089557 Best loss: 0.089557 Accuracy: 97.54% 1 Validation loss: 0.087207 Best loss: 0.087207 Accuracy: 97.85% 2 Validation loss: 0.072845 Best loss: 0.072845 Accuracy: 98.20% 3 Validation loss: 0.095476 Best loss: 0.072845 Accuracy: 97.81% 4 Validation loss: 0.079110 Best loss: 0.072845 Accuracy: 98.12% 5 Validation loss: 0.100395 Best loss: 0.072845 Accuracy: 97.93% 6 Validation loss: 0.075712 Best loss: 0.072845 Accuracy: 98.24% 7 Validation loss: 0.068755 Best loss: 0.068755 Accuracy: 98.51% 8 Validation loss: 0.060472 Best loss: 0.060472 Accuracy: 98.51% 9 Validation loss: 0.061686 Best loss: 0.060472 Accuracy: 98.51% 10 Validation loss: 0.073987 Best loss: 0.060472 Accuracy: 98.32% 11 Validation loss: 0.051687 Best loss: 0.051687 Accuracy: 98.44% 12 Validation loss: 0.076320 Best loss: 0.051687 Accuracy: 98.44% 13 Validation loss: 0.086919 Best loss: 0.051687 Accuracy: 98.12% 14 Validation loss: 0.094747 Best loss: 0.051687 Accuracy: 97.69% 15 Validation loss: 0.079594 Best loss: 0.051687 Accuracy: 98.12% 16 Validation loss: 0.067262 Best loss: 0.051687 Accuracy: 98.32% 17 Validation loss: 0.056834 Best loss: 0.051687 Accuracy: 98.55% 18 Validation loss: 0.063618 Best loss: 0.051687 Accuracy: 98.63% 19 Validation loss: 0.056871 Best loss: 0.051687 Accuracy: 98.67% 20 Validation loss: 0.057569 Best loss: 0.051687 Accuracy: 98.51% 21 Validation loss: 0.064148 Best loss: 0.051687 Accuracy: 98.63% 22 Validation loss: 0.097419 Best loss: 0.051687 Accuracy: 98.36% 23 Validation loss: 0.093054 Best loss: 0.051687 Accuracy: 98.01% 24 Validation loss: 0.058707 Best loss: 0.051687 Accuracy: 98.44% 25 Validation loss: 0.165392 Best loss: 0.051687 Accuracy: 97.69% 26 Validation loss: 0.069909 Best loss: 0.051687 Accuracy: 98.08% 27 Validation loss: 0.088141 Best loss: 0.051687 Accuracy: 98.48% 28 Validation loss: 0.067231 Best loss: 0.051687 Accuracy: 98.55% 29 Validation loss: 0.063807 Best loss: 0.051687 Accuracy: 98.55% 30 Validation loss: 0.055569 Best loss: 0.051687 Accuracy: 98.67% 31 Validation loss: 0.083008 Best loss: 0.051687 Accuracy: 98.36% 32 Validation loss: 0.068580 Best loss: 0.051687 Accuracy: 98.51% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 22.6s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2 0 Validation loss: 0.099743 Best loss: 0.099743 Accuracy: 97.34% 1 Validation loss: 0.091605 Best loss: 0.091605 Accuracy: 97.81% 2 Validation loss: 0.092293 Best loss: 0.091605 Accuracy: 98.05% 3 Validation loss: 0.070359 Best loss: 0.070359 Accuracy: 98.28% 4 Validation loss: 0.076601 Best loss: 0.070359 Accuracy: 98.05% 5 Validation loss: 0.076420 Best loss: 0.070359 Accuracy: 98.16% 6 Validation loss: 0.075343 Best loss: 0.070359 Accuracy: 98.44% 7 Validation loss: 0.064467 Best loss: 0.064467 Accuracy: 98.48% 8 Validation loss: 0.069713 Best loss: 0.064467 Accuracy: 98.40% 9 Validation loss: 0.070280 Best loss: 0.064467 Accuracy: 98.28% 10 Validation loss: 0.063510 Best loss: 0.063510 Accuracy: 98.55% 11 Validation loss: 0.083768 Best loss: 0.063510 Accuracy: 98.24% 12 Validation loss: 0.079313 Best loss: 0.063510 Accuracy: 97.69% 13 Validation loss: 0.074002 Best loss: 0.063510 Accuracy: 98.48% 14 Validation loss: 0.089146 Best loss: 0.063510 Accuracy: 98.55% 15 Validation loss: 0.085426 Best loss: 0.063510 Accuracy: 98.12% 16 Validation loss: 0.068399 Best loss: 0.063510 Accuracy: 98.71% 17 Validation loss: 0.059158 Best loss: 0.059158 Accuracy: 98.28% 18 Validation loss: 0.067200 Best loss: 0.059158 Accuracy: 98.48% 19 Validation loss: 0.076494 Best loss: 0.059158 Accuracy: 98.63% 20 Validation loss: 0.077016 Best loss: 0.059158 Accuracy: 98.40% 21 Validation loss: 0.057227 Best loss: 0.057227 Accuracy: 98.59% 22 Validation loss: 0.128858 Best loss: 0.057227 Accuracy: 98.08% 23 Validation loss: 0.081768 Best loss: 0.057227 Accuracy: 98.44% 24 Validation loss: 0.416116 Best loss: 0.057227 Accuracy: 91.87% 25 Validation loss: 0.101783 Best loss: 0.057227 Accuracy: 97.89% 26 Validation loss: 0.108605 Best loss: 0.057227 Accuracy: 98.01% 27 Validation loss: 0.088553 Best loss: 0.057227 Accuracy: 98.20% 28 Validation loss: 0.083162 Best loss: 0.057227 Accuracy: 98.16% 29 Validation loss: 0.088182 Best loss: 0.057227 Accuracy: 98.12% 30 Validation loss: 0.073804 Best loss: 0.057227 Accuracy: 98.32% 31 Validation loss: 0.085718 Best loss: 0.057227 Accuracy: 98.20% 32 Validation loss: 0.076619 Best loss: 0.057227 Accuracy: 98.24% 33 Validation loss: 0.086118 Best loss: 0.057227 Accuracy: 98.40% 34 Validation loss: 0.069324 Best loss: 0.057227 Accuracy: 98.48% 35 Validation loss: 0.088260 Best loss: 0.057227 Accuracy: 98.36% 36 Validation loss: 0.083458 Best loss: 0.057227 Accuracy: 98.40% 37 Validation loss: 0.079296 Best loss: 0.057227 Accuracy: 98.55% 38 Validation loss: 0.092029 Best loss: 0.057227 Accuracy: 98.44% 39 Validation loss: 0.072950 Best loss: 0.057227 Accuracy: 98.51% 40 Validation loss: 0.073801 Best loss: 0.057227 Accuracy: 98.59% 41 Validation loss: 0.074205 Best loss: 0.057227 Accuracy: 98.51% 42 Validation loss: 0.070172 Best loss: 0.057227 Accuracy: 98.59% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, n_neurons=70, batch_size=100, learning_rate=0.01, dropout_rate=0.2, total= 29.2s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.178649 Best loss: 0.178649 Accuracy: 95.58% 1 Validation loss: 0.177173 Best loss: 0.177173 Accuracy: 95.27% 2 Validation loss: 0.188105 Best loss: 0.177173 Accuracy: 94.80% 3 Validation loss: 0.379033 Best loss: 0.177173 Accuracy: 78.11% 4 Validation loss: 0.172508 Best loss: 0.172508 Accuracy: 95.43% 5 Validation loss: 0.162357 Best loss: 0.162357 Accuracy: 95.66% 6 Validation loss: 0.222062 Best loss: 0.162357 Accuracy: 94.96% 7 Validation loss: 0.257322 Best loss: 0.162357 Accuracy: 93.51% 8 Validation loss: 0.345270 Best loss: 0.162357 Accuracy: 90.77% 9 Validation loss: 0.288998 Best loss: 0.162357 Accuracy: 93.59% 10 Validation loss: 0.304907 Best loss: 0.162357 Accuracy: 89.87% 11 Validation loss: 0.373477 Best loss: 0.162357 Accuracy: 88.86% 12 Validation loss: 0.345029 Best loss: 0.162357 Accuracy: 89.33% 13 Validation loss: 0.312747 Best loss: 0.162357 Accuracy: 88.70% 14 Validation loss: 0.351115 Best loss: 0.162357 Accuracy: 89.17% 15 Validation loss: 0.267792 Best loss: 0.162357 Accuracy: 92.53% 16 Validation loss: 0.306894 Best loss: 0.162357 Accuracy: 91.16% 17 Validation loss: 0.310936 Best loss: 0.162357 Accuracy: 90.54% 18 Validation loss: 0.302724 Best loss: 0.162357 Accuracy: 92.03% 19 Validation loss: 0.369541 Best loss: 0.162357 Accuracy: 90.30% 20 Validation loss: 0.342737 Best loss: 0.162357 Accuracy: 88.35% 21 Validation loss: 0.333216 Best loss: 0.162357 Accuracy: 87.92% 22 Validation loss: 0.285426 Best loss: 0.162357 Accuracy: 90.15% 23 Validation loss: 0.310368 Best loss: 0.162357 Accuracy: 87.72% 24 Validation loss: 0.401841 Best loss: 0.162357 Accuracy: 84.75% 25 Validation loss: 0.341868 Best loss: 0.162357 Accuracy: 86.98% 26 Validation loss: 0.254305 Best loss: 0.162357 Accuracy: 91.44% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 30.1s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.203658 Best loss: 0.203658 Accuracy: 95.50% 1 Validation loss: 0.175531 Best loss: 0.175531 Accuracy: 94.80% 2 Validation loss: 0.177419 Best loss: 0.175531 Accuracy: 95.47% 3 Validation loss: 0.180177 Best loss: 0.175531 Accuracy: 94.68% 4 Validation loss: 0.199730 Best loss: 0.175531 Accuracy: 95.11% 5 Validation loss: 0.233501 Best loss: 0.175531 Accuracy: 93.39% 6 Validation loss: 0.153021 Best loss: 0.153021 Accuracy: 95.23% 7 Validation loss: 0.187916 Best loss: 0.153021 Accuracy: 94.61% 8 Validation loss: 0.190937 Best loss: 0.153021 Accuracy: 94.84% 9 Validation loss: 0.246666 Best loss: 0.153021 Accuracy: 90.85% 10 Validation loss: 0.518441 Best loss: 0.153021 Accuracy: 85.93% 11 Validation loss: 0.306135 Best loss: 0.153021 Accuracy: 91.99% 12 Validation loss: 0.307987 Best loss: 0.153021 Accuracy: 91.28% 13 Validation loss: 0.259230 Best loss: 0.153021 Accuracy: 91.40% 14 Validation loss: 0.344747 Best loss: 0.153021 Accuracy: 89.52% 15 Validation loss: 0.321753 Best loss: 0.153021 Accuracy: 90.07% 16 Validation loss: 0.347784 Best loss: 0.153021 Accuracy: 89.44% 17 Validation loss: 0.478495 Best loss: 0.153021 Accuracy: 82.02% 18 Validation loss: 0.316726 Best loss: 0.153021 Accuracy: 89.09% 19 Validation loss: 0.332411 Best loss: 0.153021 Accuracy: 87.84% 20 Validation loss: 0.326442 Best loss: 0.153021 Accuracy: 85.89% 21 Validation loss: 0.270364 Best loss: 0.153021 Accuracy: 92.69% 22 Validation loss: 0.297611 Best loss: 0.153021 Accuracy: 89.72% 23 Validation loss: 0.386592 Best loss: 0.153021 Accuracy: 86.79% 24 Validation loss: 0.369710 Best loss: 0.153021 Accuracy: 90.42% 25 Validation loss: 0.404215 Best loss: 0.153021 Accuracy: 88.47% 26 Validation loss: 0.544745 Best loss: 0.153021 Accuracy: 82.56% 27 Validation loss: 0.403404 Best loss: 0.153021 Accuracy: 87.80% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 31.5s [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5 0 Validation loss: 0.217006 Best loss: 0.217006 Accuracy: 94.80% 1 Validation loss: 0.215286 Best loss: 0.215286 Accuracy: 94.68% 2 Validation loss: 0.210978 Best loss: 0.210978 Accuracy: 94.06% 3 Validation loss: 0.177708 Best loss: 0.177708 Accuracy: 94.92% 4 Validation loss: 0.174023 Best loss: 0.174023 Accuracy: 95.27% 5 Validation loss: 0.244503 Best loss: 0.174023 Accuracy: 92.65% 6 Validation loss: 0.205236 Best loss: 0.174023 Accuracy: 95.86% 7 Validation loss: 0.222560 Best loss: 0.174023 Accuracy: 93.90% 8 Validation loss: 0.188206 Best loss: 0.174023 Accuracy: 94.29% 9 Validation loss: 0.201887 Best loss: 0.174023 Accuracy: 95.00% 10 Validation loss: 0.231990 Best loss: 0.174023 Accuracy: 92.92% 11 Validation loss: 0.302553 Best loss: 0.174023 Accuracy: 92.22% 12 Validation loss: 0.338830 Best loss: 0.174023 Accuracy: 91.40% 13 Validation loss: 0.943957 Best loss: 0.174023 Accuracy: 69.35% 14 Validation loss: 0.260958 Best loss: 0.174023 Accuracy: 93.24% 15 Validation loss: 0.288133 Best loss: 0.174023 Accuracy: 92.89% 16 Validation loss: 0.277220 Best loss: 0.174023 Accuracy: 91.67% 17 Validation loss: 0.260509 Best loss: 0.174023 Accuracy: 92.65% 18 Validation loss: 0.358148 Best loss: 0.174023 Accuracy: 90.30% 19 Validation loss: 0.329294 Best loss: 0.174023 Accuracy: 89.33% 20 Validation loss: 0.236151 Best loss: 0.174023 Accuracy: 92.81% 21 Validation loss: 0.317484 Best loss: 0.174023 Accuracy: 93.16% 22 Validation loss: 0.398677 Best loss: 0.174023 Accuracy: 88.31% 23 Validation loss: 0.268931 Best loss: 0.174023 Accuracy: 92.69% 24 Validation loss: 0.322406 Best loss: 0.174023 Accuracy: 90.15% 25 Validation loss: 0.255432 Best loss: 0.174023 Accuracy: 92.26% Early stopping! [CV] activation=<function relu at 0x7f95d7315f28>, n_neurons=160, batch_size=50, learning_rate=0.01, dropout_rate=0.5, total= 29.1s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 157729.093750 Best loss: 157729.093750 Accuracy: 18.73% 1 Validation loss: 131291.687500 Best loss: 131291.687500 Accuracy: 18.73% 2 Validation loss: 513699.687500 Best loss: 131291.687500 Accuracy: 18.73% 3 Validation loss: 102134.664062 Best loss: 102134.664062 Accuracy: 22.01% 4 Validation loss: 196309.609375 Best loss: 102134.664062 Accuracy: 22.01% 5 Validation loss: 624584.812500 Best loss: 102134.664062 Accuracy: 19.27% 6 Validation loss: 110465.867188 Best loss: 102134.664062 Accuracy: 22.01% 7 Validation loss: 48578.433594 Best loss: 48578.433594 Accuracy: 18.73% 8 Validation loss: 21699.724609 Best loss: 21699.724609 Accuracy: 18.73% 9 Validation loss: 42311.808594 Best loss: 21699.724609 Accuracy: 22.01% 10 Validation loss: 36158.656250 Best loss: 21699.724609 Accuracy: 19.27% 11 Validation loss: 25769.384766 Best loss: 21699.724609 Accuracy: 18.73% 12 Validation loss: 27089.314453 Best loss: 21699.724609 Accuracy: 19.27% 13 Validation loss: 9064.200195 Best loss: 9064.200195 Accuracy: 19.27% 14 Validation loss: 8311.001953 Best loss: 8311.001953 Accuracy: 19.08% 15 Validation loss: 4842.810547 Best loss: 4842.810547 Accuracy: 18.73% 16 Validation loss: 28856.513672 Best loss: 4842.810547 Accuracy: 18.73% 17 Validation loss: 3669.973389 Best loss: 3669.973389 Accuracy: 18.73% 18 Validation loss: 15393.154297 Best loss: 3669.973389 Accuracy: 22.01% 19 Validation loss: 4523.440430 Best loss: 3669.973389 Accuracy: 18.73% 20 Validation loss: 571963.625000 Best loss: 3669.973389 Accuracy: 22.01% 21 Validation loss: 105328.703125 Best loss: 3669.973389 Accuracy: 22.01% 22 Validation loss: 14451.157227 Best loss: 3669.973389 Accuracy: 18.73% 23 Validation loss: 20326.818359 Best loss: 3669.973389 Accuracy: 22.01% 24 Validation loss: 6813.783203 Best loss: 3669.973389 Accuracy: 20.91% 25 Validation loss: 251909.078125 Best loss: 3669.973389 Accuracy: 22.01% 26 Validation loss: 70195.757812 Best loss: 3669.973389 Accuracy: 20.91% 27 Validation loss: 16774.457031 Best loss: 3669.973389 Accuracy: 22.01% 28 Validation loss: 146904.015625 Best loss: 3669.973389 Accuracy: 20.91% 29 Validation loss: 5349.644531 Best loss: 3669.973389 Accuracy: 20.91% 30 Validation loss: 72969.257812 Best loss: 3669.973389 Accuracy: 18.73% 31 Validation loss: 7125.093262 Best loss: 3669.973389 Accuracy: 18.73% 32 Validation loss: 7618.934082 Best loss: 3669.973389 Accuracy: 19.08% 33 Validation loss: 1809.641479 Best loss: 1809.641479 Accuracy: 20.91% 34 Validation loss: 1543.906738 Best loss: 1543.906738 Accuracy: 19.27% 35 Validation loss: 2308.916260 Best loss: 1543.906738 Accuracy: 19.08% 36 Validation loss: 4926.201172 Best loss: 1543.906738 Accuracy: 19.27% 37 Validation loss: 2614.192383 Best loss: 1543.906738 Accuracy: 18.88% 38 Validation loss: 1284.252197 Best loss: 1284.252197 Accuracy: 18.73% 39 Validation loss: 775.029907 Best loss: 775.029907 Accuracy: 20.56% 40 Validation loss: 493.371002 Best loss: 493.371002 Accuracy: 20.91% 41 Validation loss: 623.600830 Best loss: 493.371002 Accuracy: 19.12% 42 Validation loss: 829.203491 Best loss: 493.371002 Accuracy: 20.91% 43 Validation loss: 1722.264404 Best loss: 493.371002 Accuracy: 20.91% 44 Validation loss: 725.053711 Best loss: 493.371002 Accuracy: 22.01% 45 Validation loss: 373.615662 Best loss: 373.615662 Accuracy: 21.15% 46 Validation loss: 18776.503906 Best loss: 373.615662 Accuracy: 19.08% 47 Validation loss: 64629.917969 Best loss: 373.615662 Accuracy: 22.01% 48 Validation loss: 23454448.000000 Best loss: 373.615662 Accuracy: 20.91% 49 Validation loss: 2295121.500000 Best loss: 373.615662 Accuracy: 31.78% 50 Validation loss: 2808697.500000 Best loss: 373.615662 Accuracy: 19.08% 51 Validation loss: 740383.312500 Best loss: 373.615662 Accuracy: 18.73% 52 Validation loss: 843701.250000 Best loss: 373.615662 Accuracy: 20.91% 53 Validation loss: 572359.000000 Best loss: 373.615662 Accuracy: 18.73% 54 Validation loss: 433033.906250 Best loss: 373.615662 Accuracy: 18.73% 55 Validation loss: 1794766.750000 Best loss: 373.615662 Accuracy: 19.08% 56 Validation loss: 483934.312500 Best loss: 373.615662 Accuracy: 20.91% 57 Validation loss: 679108.187500 Best loss: 373.615662 Accuracy: 19.08% 58 Validation loss: 206286.906250 Best loss: 373.615662 Accuracy: 20.91% 59 Validation loss: 623736.187500 Best loss: 373.615662 Accuracy: 22.01% 60 Validation loss: 171049.812500 Best loss: 373.615662 Accuracy: 18.73% 61 Validation loss: 416614.187500 Best loss: 373.615662 Accuracy: 19.08% 62 Validation loss: 405088.500000 Best loss: 373.615662 Accuracy: 19.27% 63 Validation loss: 132533.171875 Best loss: 373.615662 Accuracy: 20.91% 64 Validation loss: 110313.335938 Best loss: 373.615662 Accuracy: 19.27% 65 Validation loss: 71637.484375 Best loss: 373.615662 Accuracy: 18.69% 66 Validation loss: 189933.890625 Best loss: 373.615662 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 44.7s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 8284.606445 Best loss: 8284.606445 Accuracy: 19.27% 1 Validation loss: 49194.476562 Best loss: 8284.606445 Accuracy: 20.91% 2 Validation loss: 38139.980469 Best loss: 8284.606445 Accuracy: 19.08% 3 Validation loss: 186506.156250 Best loss: 8284.606445 Accuracy: 18.73% 4 Validation loss: 60522.125000 Best loss: 8284.606445 Accuracy: 18.73% 5 Validation loss: 26706.923828 Best loss: 8284.606445 Accuracy: 20.91% 6 Validation loss: 26576.337891 Best loss: 8284.606445 Accuracy: 19.31% 7 Validation loss: 2990.574951 Best loss: 2990.574951 Accuracy: 19.08% 8 Validation loss: 8886.087891 Best loss: 2990.574951 Accuracy: 19.27% 9 Validation loss: 5703.571777 Best loss: 2990.574951 Accuracy: 22.01% 10 Validation loss: 2621.260498 Best loss: 2621.260498 Accuracy: 19.08% 11 Validation loss: 1005.041748 Best loss: 1005.041748 Accuracy: 19.08% 12 Validation loss: 2834.614258 Best loss: 1005.041748 Accuracy: 20.91% 13 Validation loss: 2647.386719 Best loss: 1005.041748 Accuracy: 19.08% 14 Validation loss: 2423.319092 Best loss: 1005.041748 Accuracy: 18.73% 15 Validation loss: 2245.293701 Best loss: 1005.041748 Accuracy: 20.91% 16 Validation loss: 819.370239 Best loss: 819.370239 Accuracy: 22.01% 17 Validation loss: 880.817444 Best loss: 819.370239 Accuracy: 18.73% 18 Validation loss: 1496.655518 Best loss: 819.370239 Accuracy: 22.01% 19 Validation loss: 1065.911499 Best loss: 819.370239 Accuracy: 19.27% 20 Validation loss: 2472.805664 Best loss: 819.370239 Accuracy: 19.08% 21 Validation loss: 5815.130371 Best loss: 819.370239 Accuracy: 19.27% 22 Validation loss: 7922.421387 Best loss: 819.370239 Accuracy: 19.08% 23 Validation loss: 3974.250244 Best loss: 819.370239 Accuracy: 20.91% 24 Validation loss: 614.218567 Best loss: 614.218567 Accuracy: 26.39% 25 Validation loss: 7579.687988 Best loss: 614.218567 Accuracy: 20.91% 26 Validation loss: 896.921814 Best loss: 614.218567 Accuracy: 18.73% 27 Validation loss: 659.835815 Best loss: 614.218567 Accuracy: 25.18% 28 Validation loss: 678.968140 Best loss: 614.218567 Accuracy: 19.08% 29 Validation loss: 961.812378 Best loss: 614.218567 Accuracy: 19.27% 30 Validation loss: 542.250305 Best loss: 542.250305 Accuracy: 19.16% 31 Validation loss: 2099.243164 Best loss: 542.250305 Accuracy: 19.08% 32 Validation loss: 2453.611084 Best loss: 542.250305 Accuracy: 20.91% 33 Validation loss: 35029104.000000 Best loss: 542.250305 Accuracy: 18.73% 34 Validation loss: 5267460.000000 Best loss: 542.250305 Accuracy: 18.73% 35 Validation loss: 4463219.000000 Best loss: 542.250305 Accuracy: 20.91% 36 Validation loss: 2119073.500000 Best loss: 542.250305 Accuracy: 22.01% 37 Validation loss: 8465989.000000 Best loss: 542.250305 Accuracy: 18.73% 38 Validation loss: 3447434.000000 Best loss: 542.250305 Accuracy: 20.91% 39 Validation loss: 3167589.000000 Best loss: 542.250305 Accuracy: 22.01% 40 Validation loss: 6869570.000000 Best loss: 542.250305 Accuracy: 18.73% 41 Validation loss: 2087190.000000 Best loss: 542.250305 Accuracy: 22.01% 42 Validation loss: 661229.500000 Best loss: 542.250305 Accuracy: 22.01% 43 Validation loss: 633635.000000 Best loss: 542.250305 Accuracy: 22.01% 44 Validation loss: 874969.187500 Best loss: 542.250305 Accuracy: 19.08% 45 Validation loss: 552155.500000 Best loss: 542.250305 Accuracy: 18.73% 46 Validation loss: 1064265.500000 Best loss: 542.250305 Accuracy: 18.73% 47 Validation loss: 547671.000000 Best loss: 542.250305 Accuracy: 18.73% 48 Validation loss: 816417.187500 Best loss: 542.250305 Accuracy: 20.91% 49 Validation loss: 497213.687500 Best loss: 542.250305 Accuracy: 22.01% 50 Validation loss: 457019.343750 Best loss: 542.250305 Accuracy: 22.01% 51 Validation loss: 869266.687500 Best loss: 542.250305 Accuracy: 20.91% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 34.9s [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6 0 Validation loss: 94741.093750 Best loss: 94741.093750 Accuracy: 20.91% 1 Validation loss: 10666.666992 Best loss: 10666.666992 Accuracy: 19.27% 2 Validation loss: 1230.552979 Best loss: 1230.552979 Accuracy: 20.91% 3 Validation loss: 2476.391846 Best loss: 1230.552979 Accuracy: 20.91% 4 Validation loss: 981.359558 Best loss: 981.359558 Accuracy: 19.08% 5 Validation loss: 22060.767578 Best loss: 981.359558 Accuracy: 19.27% 6 Validation loss: 10318.696289 Best loss: 981.359558 Accuracy: 19.08% 7 Validation loss: 3932.526123 Best loss: 981.359558 Accuracy: 19.27% 8 Validation loss: 12767560.000000 Best loss: 981.359558 Accuracy: 19.08% 9 Validation loss: 4181907.000000 Best loss: 981.359558 Accuracy: 22.01% 10 Validation loss: 1951308.000000 Best loss: 981.359558 Accuracy: 18.73% 11 Validation loss: 1822542.875000 Best loss: 981.359558 Accuracy: 19.27% 12 Validation loss: 790750.375000 Best loss: 981.359558 Accuracy: 20.91% 13 Validation loss: 370165.750000 Best loss: 981.359558 Accuracy: 19.27% 14 Validation loss: 744099.750000 Best loss: 981.359558 Accuracy: 19.27% 15 Validation loss: 401977.031250 Best loss: 981.359558 Accuracy: 19.08% 16 Validation loss: 478552.281250 Best loss: 981.359558 Accuracy: 19.27% 17 Validation loss: 305866.343750 Best loss: 981.359558 Accuracy: 19.08% 18 Validation loss: 166979.234375 Best loss: 981.359558 Accuracy: 20.91% 19 Validation loss: 67455.671875 Best loss: 981.359558 Accuracy: 20.91% 20 Validation loss: 94393.250000 Best loss: 981.359558 Accuracy: 18.73% 21 Validation loss: 156004.421875 Best loss: 981.359558 Accuracy: 19.08% 22 Validation loss: 142700.812500 Best loss: 981.359558 Accuracy: 19.27% 23 Validation loss: 44230.230469 Best loss: 981.359558 Accuracy: 18.73% 24 Validation loss: 49378.320312 Best loss: 981.359558 Accuracy: 18.73% 25 Validation loss: 1947654.250000 Best loss: 981.359558 Accuracy: 19.08% Early stopping! [CV] activation=<function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>, n_neurons=100, batch_size=100, learning_rate=0.1, dropout_rate=0.6, total= 17.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 1.780113 Best loss: 1.780113 Accuracy: 19.27% 1 Validation loss: 1.846068 Best loss: 1.780113 Accuracy: 19.08% 2 Validation loss: 1.852679 Best loss: 1.780113 Accuracy: 19.08% 3 Validation loss: 1.670971 Best loss: 1.670971 Accuracy: 22.01% 4 Validation loss: 1.628554 Best loss: 1.628554 Accuracy: 18.73% 5 Validation loss: 1.629447 Best loss: 1.628554 Accuracy: 22.01% 6 Validation loss: 1.922497 Best loss: 1.628554 Accuracy: 18.73% 7 Validation loss: 1.731570 Best loss: 1.628554 Accuracy: 19.27% 8 Validation loss: 1.752681 Best loss: 1.628554 Accuracy: 19.27% 9 Validation loss: 1.744861 Best loss: 1.628554 Accuracy: 19.27% 10 Validation loss: 1.681446 Best loss: 1.628554 Accuracy: 20.91% 11 Validation loss: 1.667080 Best loss: 1.628554 Accuracy: 19.08% 12 Validation loss: 1.699583 Best loss: 1.628554 Accuracy: 19.08% 13 Validation loss: 1.815837 Best loss: 1.628554 Accuracy: 19.27% 14 Validation loss: 1.790249 Best loss: 1.628554 Accuracy: 18.73% 15 Validation loss: 1.792745 Best loss: 1.628554 Accuracy: 22.01% 16 Validation loss: 1.838525 Best loss: 1.628554 Accuracy: 22.01% 17 Validation loss: 1.650854 Best loss: 1.628554 Accuracy: 22.01% 18 Validation loss: 2.190966 Best loss: 1.628554 Accuracy: 19.27% 19 Validation loss: 1.804792 Best loss: 1.628554 Accuracy: 20.91% 20 Validation loss: 1.626028 Best loss: 1.626028 Accuracy: 22.01% 21 Validation loss: 1.725462 Best loss: 1.626028 Accuracy: 19.27% 22 Validation loss: 1.638565 Best loss: 1.626028 Accuracy: 19.08% 23 Validation loss: 1.902902 Best loss: 1.626028 Accuracy: 22.01% 24 Validation loss: 1.670607 Best loss: 1.626028 Accuracy: 20.91% 25 Validation loss: 1.848546 Best loss: 1.626028 Accuracy: 19.27% 26 Validation loss: 1.730690 Best loss: 1.626028 Accuracy: 19.08% 27 Validation loss: 1.771860 Best loss: 1.626028 Accuracy: 22.01% 28 Validation loss: 1.745714 Best loss: 1.626028 Accuracy: 19.08% 29 Validation loss: 1.753474 Best loss: 1.626028 Accuracy: 18.73% 30 Validation loss: 1.712327 Best loss: 1.626028 Accuracy: 18.73% 31 Validation loss: 1.717302 Best loss: 1.626028 Accuracy: 22.01% 32 Validation loss: 1.806233 Best loss: 1.626028 Accuracy: 18.73% 33 Validation loss: 1.837658 Best loss: 1.626028 Accuracy: 22.01% 34 Validation loss: 1.676526 Best loss: 1.626028 Accuracy: 22.01% 35 Validation loss: 1.865379 Best loss: 1.626028 Accuracy: 19.08% 36 Validation loss: 1.858150 Best loss: 1.626028 Accuracy: 22.01% 37 Validation loss: 1.888744 Best loss: 1.626028 Accuracy: 19.27% 38 Validation loss: 1.845071 Best loss: 1.626028 Accuracy: 18.73% 39 Validation loss: 1.673971 Best loss: 1.626028 Accuracy: 19.08% 40 Validation loss: 1.703943 Best loss: 1.626028 Accuracy: 19.27% 41 Validation loss: 1.875259 Best loss: 1.626028 Accuracy: 20.91% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5, total= 25.1s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 1.903882 Best loss: 1.903882 Accuracy: 20.91% 1 Validation loss: 1.877601 Best loss: 1.877601 Accuracy: 19.08% 2 Validation loss: 1.695655 Best loss: 1.695655 Accuracy: 20.91% 3 Validation loss: 1.677254 Best loss: 1.677254 Accuracy: 22.01% 4 Validation loss: 1.672214 Best loss: 1.672214 Accuracy: 20.91% 5 Validation loss: 1.644411 Best loss: 1.644411 Accuracy: 19.08% 6 Validation loss: 1.713753 Best loss: 1.644411 Accuracy: 19.27% 7 Validation loss: 1.826744 Best loss: 1.644411 Accuracy: 18.73% 8 Validation loss: 1.666898 Best loss: 1.644411 Accuracy: 18.73% 9 Validation loss: 2.062810 Best loss: 1.644411 Accuracy: 19.08% 10 Validation loss: 1.724004 Best loss: 1.644411 Accuracy: 20.91% 11 Validation loss: 1.704440 Best loss: 1.644411 Accuracy: 22.01% 12 Validation loss: 1.637996 Best loss: 1.637996 Accuracy: 20.91% 13 Validation loss: 1.626474 Best loss: 1.626474 Accuracy: 19.08% 14 Validation loss: 1.873336 Best loss: 1.626474 Accuracy: 19.27% 15 Validation loss: 1.632297 Best loss: 1.626474 Accuracy: 22.01% 16 Validation loss: 1.720470 Best loss: 1.626474 Accuracy: 22.01% 17 Validation loss: 1.755978 Best loss: 1.626474 Accuracy: 19.08% 18 Validation loss: 1.795167 Best loss: 1.626474 Accuracy: 19.27% 19 Validation loss: 1.734717 Best loss: 1.626474 Accuracy: 19.08% 20 Validation loss: 1.783388 Best loss: 1.626474 Accuracy: 19.08% 21 Validation loss: 1.821460 Best loss: 1.626474 Accuracy: 22.01% 22 Validation loss: 1.862568 Best loss: 1.626474 Accuracy: 19.08% 23 Validation loss: 1.694009 Best loss: 1.626474 Accuracy: 20.91% 24 Validation loss: 2.257699 Best loss: 1.626474 Accuracy: 18.73% 25 Validation loss: 1.842749 Best loss: 1.626474 Accuracy: 22.01% 26 Validation loss: 1.737417 Best loss: 1.626474 Accuracy: 19.08% 27 Validation loss: 1.655710 Best loss: 1.626474 Accuracy: 20.91% 28 Validation loss: 2.259833 Best loss: 1.626474 Accuracy: 19.08% 29 Validation loss: 1.735183 Best loss: 1.626474 Accuracy: 19.08% 30 Validation loss: 1.833456 Best loss: 1.626474 Accuracy: 18.73% 31 Validation loss: 2.038689 Best loss: 1.626474 Accuracy: 22.01% 32 Validation loss: 1.925897 Best loss: 1.626474 Accuracy: 19.08% 33 Validation loss: 1.745488 Best loss: 1.626474 Accuracy: 18.73% 34 Validation loss: 1.660741 Best loss: 1.626474 Accuracy: 19.27% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5, total= 20.9s [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5 0 Validation loss: 1.911067 Best loss: 1.911067 Accuracy: 20.91% 1 Validation loss: 1.749591 Best loss: 1.749591 Accuracy: 19.08% 2 Validation loss: 1.694123 Best loss: 1.694123 Accuracy: 20.91% 3 Validation loss: 1.694709 Best loss: 1.694123 Accuracy: 19.27% 4 Validation loss: 1.675722 Best loss: 1.675722 Accuracy: 20.91% 5 Validation loss: 1.890161 Best loss: 1.675722 Accuracy: 20.91% 6 Validation loss: 1.997274 Best loss: 1.675722 Accuracy: 19.27% 7 Validation loss: 1.892825 Best loss: 1.675722 Accuracy: 18.73% 8 Validation loss: 1.680913 Best loss: 1.675722 Accuracy: 19.27% 9 Validation loss: 1.700945 Best loss: 1.675722 Accuracy: 19.08% 10 Validation loss: 1.671868 Best loss: 1.671868 Accuracy: 19.08% 11 Validation loss: 1.830365 Best loss: 1.671868 Accuracy: 22.01% 12 Validation loss: 1.750594 Best loss: 1.671868 Accuracy: 19.08% 13 Validation loss: 1.663442 Best loss: 1.663442 Accuracy: 22.01% 14 Validation loss: 2.042338 Best loss: 1.663442 Accuracy: 22.01% 15 Validation loss: 1.684646 Best loss: 1.663442 Accuracy: 22.01% 16 Validation loss: 1.848600 Best loss: 1.663442 Accuracy: 22.01% 17 Validation loss: 1.684331 Best loss: 1.663442 Accuracy: 18.73% 18 Validation loss: 1.611484 Best loss: 1.611484 Accuracy: 22.01% 19 Validation loss: 1.849918 Best loss: 1.611484 Accuracy: 22.01% 20 Validation loss: 1.779999 Best loss: 1.611484 Accuracy: 19.27% 21 Validation loss: 1.722875 Best loss: 1.611484 Accuracy: 22.01% 22 Validation loss: 1.812561 Best loss: 1.611484 Accuracy: 20.91% 23 Validation loss: 1.666043 Best loss: 1.611484 Accuracy: 20.91% 24 Validation loss: 1.826547 Best loss: 1.611484 Accuracy: 18.73% 25 Validation loss: 1.703637 Best loss: 1.611484 Accuracy: 22.01% 26 Validation loss: 1.697396 Best loss: 1.611484 Accuracy: 18.73% 27 Validation loss: 1.699131 Best loss: 1.611484 Accuracy: 20.91% 28 Validation loss: 2.069793 Best loss: 1.611484 Accuracy: 19.08% 29 Validation loss: 1.889511 Best loss: 1.611484 Accuracy: 19.08% 30 Validation loss: 1.888257 Best loss: 1.611484 Accuracy: 20.91% 31 Validation loss: 1.919254 Best loss: 1.611484 Accuracy: 22.01% 32 Validation loss: 1.716114 Best loss: 1.611484 Accuracy: 19.08% 33 Validation loss: 1.656737 Best loss: 1.611484 Accuracy: 19.08% 34 Validation loss: 1.764443 Best loss: 1.611484 Accuracy: 20.91% 35 Validation loss: 1.671702 Best loss: 1.611484 Accuracy: 22.01% 36 Validation loss: 1.640811 Best loss: 1.611484 Accuracy: 20.91% 37 Validation loss: 1.667034 Best loss: 1.611484 Accuracy: 22.01% 38 Validation loss: 1.856114 Best loss: 1.611484 Accuracy: 18.73% 39 Validation loss: 1.651554 Best loss: 1.611484 Accuracy: 19.08% Early stopping! [CV] activation=<function elu at 0x7f95d738dbf8>, n_neurons=140, batch_size=100, learning_rate=0.05, dropout_rate=0.5, total= 23.6s
[Parallel(n_jobs=1)]: Done 150 out of 150 | elapsed: 148.8min finished
0 Validation loss: 0.113959 Best loss: 0.113959 Accuracy: 97.22% 1 Validation loss: 0.085518 Best loss: 0.085518 Accuracy: 97.89% 2 Validation loss: 0.070744 Best loss: 0.070744 Accuracy: 98.05% 3 Validation loss: 0.077869 Best loss: 0.070744 Accuracy: 98.08% 4 Validation loss: 0.086914 Best loss: 0.070744 Accuracy: 97.81% 5 Validation loss: 0.063190 Best loss: 0.063190 Accuracy: 98.12% 6 Validation loss: 0.059813 Best loss: 0.059813 Accuracy: 98.51% 7 Validation loss: 0.063335 Best loss: 0.059813 Accuracy: 98.36% 8 Validation loss: 0.067481 Best loss: 0.059813 Accuracy: 98.08% 9 Validation loss: 0.068965 Best loss: 0.059813 Accuracy: 98.44% 10 Validation loss: 0.060383 Best loss: 0.059813 Accuracy: 98.55% 11 Validation loss: 0.074622 Best loss: 0.059813 Accuracy: 98.44% 12 Validation loss: 0.071615 Best loss: 0.059813 Accuracy: 98.51% 13 Validation loss: 0.051031 Best loss: 0.051031 Accuracy: 98.51% 14 Validation loss: 0.081497 Best loss: 0.051031 Accuracy: 98.40% 15 Validation loss: 0.073281 Best loss: 0.051031 Accuracy: 98.40% 16 Validation loss: 0.064080 Best loss: 0.051031 Accuracy: 98.48% 17 Validation loss: 0.048973 Best loss: 0.048973 Accuracy: 98.59% 18 Validation loss: 0.053533 Best loss: 0.048973 Accuracy: 98.59% 19 Validation loss: 0.071903 Best loss: 0.048973 Accuracy: 98.40% 20 Validation loss: 0.059117 Best loss: 0.048973 Accuracy: 98.59% 21 Validation loss: 0.046934 Best loss: 0.046934 Accuracy: 98.67% 22 Validation loss: 0.053639 Best loss: 0.046934 Accuracy: 98.24% 23 Validation loss: 0.073301 Best loss: 0.046934 Accuracy: 98.44% 24 Validation loss: 0.080359 Best loss: 0.046934 Accuracy: 98.08% 25 Validation loss: 0.056793 Best loss: 0.046934 Accuracy: 98.28% 26 Validation loss: 0.066187 Best loss: 0.046934 Accuracy: 98.20% 27 Validation loss: 0.173657 Best loss: 0.046934 Accuracy: 98.01% 28 Validation loss: 0.071967 Best loss: 0.046934 Accuracy: 98.48% 29 Validation loss: 0.130565 Best loss: 0.046934 Accuracy: 97.22% 30 Validation loss: 0.109358 Best loss: 0.046934 Accuracy: 97.77% 31 Validation loss: 0.081005 Best loss: 0.046934 Accuracy: 98.36% 32 Validation loss: 0.112478 Best loss: 0.046934 Accuracy: 98.40% 33 Validation loss: 0.122881 Best loss: 0.046934 Accuracy: 97.89% 34 Validation loss: 0.077677 Best loss: 0.046934 Accuracy: 98.79% 35 Validation loss: 0.077727 Best loss: 0.046934 Accuracy: 98.40% 36 Validation loss: 0.055566 Best loss: 0.046934 Accuracy: 98.32% 37 Validation loss: 0.096077 Best loss: 0.046934 Accuracy: 98.08% 38 Validation loss: 0.064077 Best loss: 0.046934 Accuracy: 98.36% 39 Validation loss: 0.073264 Best loss: 0.046934 Accuracy: 98.32% 40 Validation loss: 0.056025 Best loss: 0.046934 Accuracy: 98.51% 41 Validation loss: 0.058697 Best loss: 0.046934 Accuracy: 98.28% 42 Validation loss: 0.071478 Best loss: 0.046934 Accuracy: 98.24% Early stopping!
Out[136]:
RandomizedSearchCV(cv=None, error_score='raise',
estimator=DNNClassifier(activation=<function elu at 0x7f95d738dbf8>,
batch_norm_momentum=None, batch_size=20, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=5, n_neurons=100,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42),
fit_params={'y_valid': array([0, 4, ..., 1, 2], dtype=int32), 'X_valid': array([[0., 0., ..., 0., 0.],
[0., 0., ..., 0., 0.],
...,
[0., 0., ..., 0., 0.],
[0., 0., ..., 0., 0.]], dtype=float32), 'n_epochs': 1000},
iid=True, n_iter=50, n_jobs=1,
param_distributions={'activation': [<function relu at 0x7f95d7315f28>, <function elu at 0x7f95d738dbf8>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c2809510>, <function leaky_relu.<locals>.parametrized_leaky_relu at 0x7f94c1fb4a60>], 'n_neurons': [10, 30, 50, 70, 90, 100, 120, 140, 160], 'batch_size': [10, 50, 100, 500], 'learning_rate': [0.01, 0.02, 0.05, 0.1], 'dropout_rate': [0.2, 0.3, 0.4, 0.5, 0.6]},
pre_dispatch='2*n_jobs', random_state=42, refit=True,
return_train_score='warn', scoring=None, verbose=2)
In [137]:
rnd_search_dropout.best_params_
Out[137]:
{'activation': <function tensorflow.python.ops.gen_nn_ops.relu(features, name=None)>,
'batch_size': 100,
'dropout_rate': 0.2,
'learning_rate': 0.01,
'n_neurons': 160}
In [138]:
y_pred = rnd_search_dropout.predict(X_test1)
accuracy_score(y_test1, y_pred)
Out[138]:
0.9904650710254913
Oh well, dropout did not improve the model. Better luck next time! :)
But that's okay, we have ourselves a nice DNN that achieves 99.40% accuracy on the test set using Batch Normalization, or 99.32% without BN. Let's see if some of this expertise on digits 0 to 4 can be transferred to the task of classifying digits 5 to 9. For the sake of simplicity we will reuse the DNN without BN, since it is almost as good.
9. Transfer learning¶
9.1.¶
Exercise: create a new DNN that reuses all the pretrained hidden layers of the previous model, freezes them, and replaces the softmax output layer with a new one.
Let's load the best model's graph and get a handle on all the important operations we will need. Note that instead of creating a new softmax output layer, we will just reuse the existing one (since it has the same number of outputs as the existing one). We will reinitialize its parameters before training.
In [139]:
reset_graph()
restore_saver = tf.train.import_meta_graph("./my_best_mnist_model_0_to_4.meta")
X = tf.get_default_graph().get_tensor_by_name("X:0")
y = tf.get_default_graph().get_tensor_by_name("y:0")
loss = tf.get_default_graph().get_tensor_by_name("loss:0")
Y_proba = tf.get_default_graph().get_tensor_by_name("Y_proba:0")
logits = Y_proba.op.inputs[0]
accuracy = tf.get_default_graph().get_tensor_by_name("accuracy:0")
To freeze the lower layers, we will exclude their variables from the optimizer's list of trainable variables, keeping only the output layer's trainable variables:
In [140]:
learning_rate = 0.01
output_layer_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope="logits")
optimizer = tf.train.AdamOptimizer(learning_rate, name="Adam2")
training_op = optimizer.minimize(loss, var_list=output_layer_vars)
In [141]:
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
init = tf.global_variables_initializer()
five_frozen_saver = tf.train.Saver()
9.2.¶
Exercise: train this new DNN on digits 5 to 9, using only 100 images per digit, and time how long it takes. Despite this small number of examples, can you achieve high precision?
Let's create the training, validation and test sets. We need to subtract 5 from the labels because TensorFlow expects integers from 0 to n_classes-1.
In [142]:
X_train2_full = X_train[y_train >= 5]
y_train2_full = y_train[y_train >= 5] - 5
X_valid2_full = X_valid[y_valid >= 5]
y_valid2_full = y_valid[y_valid >= 5] - 5
X_test2 = X_test[y_test >= 5]
y_test2 = y_test[y_test >= 5] - 5
Also, for the purpose of this exercise, we want to keep only 100 instances per class in the training set (and let's keep only 30 instances per class in the validation set). Let's create a small function to do that:
In [143]:
def sample_n_instances_per_class(X, y, n=100):
Xs, ys = [], []
for label in np.unique(y):
idx = (y == label)
Xc = X[idx][:n]
yc = y[idx][:n]
Xs.append(Xc)
ys.append(yc)
return np.concatenate(Xs), np.concatenate(ys)
In [144]:
X_train2, y_train2 = sample_n_instances_per_class(X_train2_full, y_train2_full, n=100)
X_valid2, y_valid2 = sample_n_instances_per_class(X_valid2_full, y_valid2_full, n=30)
Now let's train the model. This is the same training code as earlier, using early stopping, except for the initialization: we first initialize all the variables, then we restore the best model trained earlier (on digits 0 to 4), and finally we reinitialize the output layer variables.
In [145]:
import time
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_best_mnist_model_0_to_4")
for var in output_layer_vars:
var.initializer.run()
t0 = time.time()
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid2, y: y_valid2})
if loss_val < best_loss:
save_path = five_frozen_saver.save(sess, "./my_mnist_model_5_to_9_five_frozen")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
t1 = time.time()
print("Total training time: {:.1f}s".format(t1 - t0))
with tf.Session() as sess:
five_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_five_frozen")
acc_test = accuracy.eval(feed_dict={X: X_test2, y: y_test2})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
INFO:tensorflow:Restoring parameters from ./my_best_mnist_model_0_to_4 0 Validation loss: 0.951825 Best loss: 0.951825 Accuracy: 69.33% 1 Validation loss: 0.893576 Best loss: 0.893576 Accuracy: 70.00% 2 Validation loss: 0.820447 Best loss: 0.820447 Accuracy: 66.00% 3 Validation loss: 0.801238 Best loss: 0.801238 Accuracy: 68.00% 4 Validation loss: 0.810730 Best loss: 0.801238 Accuracy: 71.33% 5 Validation loss: 0.849664 Best loss: 0.801238 Accuracy: 67.33% 6 Validation loss: 0.759985 Best loss: 0.759985 Accuracy: 73.33% 7 Validation loss: 0.741326 Best loss: 0.741326 Accuracy: 76.00% 8 Validation loss: 0.788584 Best loss: 0.741326 Accuracy: 71.33% 9 Validation loss: 0.722393 Best loss: 0.722393 Accuracy: 75.33% 10 Validation loss: 0.862122 Best loss: 0.722393 Accuracy: 69.33% 11 Validation loss: 0.737343 Best loss: 0.722393 Accuracy: 73.33% 12 Validation loss: 0.730194 Best loss: 0.722393 Accuracy: 73.33% 13 Validation loss: 0.792437 Best loss: 0.722393 Accuracy: 70.67% 14 Validation loss: 0.941767 Best loss: 0.722393 Accuracy: 68.67% 15 Validation loss: 0.790877 Best loss: 0.722393 Accuracy: 70.67% 16 Validation loss: 0.868693 Best loss: 0.722393 Accuracy: 69.33% 17 Validation loss: 0.835207 Best loss: 0.722393 Accuracy: 72.00% 18 Validation loss: 0.701575 Best loss: 0.701575 Accuracy: 74.67% 19 Validation loss: 0.647492 Best loss: 0.647492 Accuracy: 76.00% 20 Validation loss: 0.742936 Best loss: 0.647492 Accuracy: 76.00% 21 Validation loss: 0.770601 Best loss: 0.647492 Accuracy: 74.67% 22 Validation loss: 0.713722 Best loss: 0.647492 Accuracy: 75.33% 23 Validation loss: 0.704974 Best loss: 0.647492 Accuracy: 74.67% 24 Validation loss: 0.725125 Best loss: 0.647492 Accuracy: 74.00% 25 Validation loss: 0.771376 Best loss: 0.647492 Accuracy: 72.67% 26 Validation loss: 0.714716 Best loss: 0.647492 Accuracy: 77.33% 27 Validation loss: 0.717239 Best loss: 0.647492 Accuracy: 76.67% 28 Validation loss: 0.783174 Best loss: 0.647492 Accuracy: 72.00% 29 Validation loss: 0.718706 Best loss: 0.647492 Accuracy: 75.33% 30 Validation loss: 0.736024 Best loss: 0.647492 Accuracy: 76.00% 31 Validation loss: 0.743669 Best loss: 0.647492 Accuracy: 74.67% 32 Validation loss: 0.717809 Best loss: 0.647492 Accuracy: 76.67% 33 Validation loss: 0.700637 Best loss: 0.647492 Accuracy: 76.00% 34 Validation loss: 0.752360 Best loss: 0.647492 Accuracy: 71.33% 35 Validation loss: 0.765944 Best loss: 0.647492 Accuracy: 74.67% 36 Validation loss: 0.780383 Best loss: 0.647492 Accuracy: 74.67% 37 Validation loss: 0.803066 Best loss: 0.647492 Accuracy: 71.33% 38 Validation loss: 0.786106 Best loss: 0.647492 Accuracy: 72.67% 39 Validation loss: 0.742891 Best loss: 0.647492 Accuracy: 77.33% Early stopping! Total training time: 2.2s INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_five_frozen Final test accuracy: 71.69%
Well that's not a great accuracy, is it? Of course with such a tiny training set, and with only one layer to tweak, we should not expect miracles.
9.3.¶
Exercise: try caching the frozen layers, and train the model again: how much faster is it now?
Let's start by getting a handle on the output of the last frozen layer:
In [146]:
hidden5_out = tf.get_default_graph().get_tensor_by_name("hidden5_out:0")
Now let's train the model using roughly the same code as earlier. The difference is that we compute the output of the top frozen layer at the beginning (both for the training set and the validation set), and we cache it. This makes training roughly 1.5 to 3 times faster in this example (this may vary greatly, depending on your system):
In [147]:
import time
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_best_mnist_model_0_to_4")
for var in output_layer_vars:
var.initializer.run()
t0 = time.time()
hidden5_train = hidden5_out.eval(feed_dict={X: X_train2, y: y_train2})
hidden5_valid = hidden5_out.eval(feed_dict={X: X_valid2, y: y_valid2})
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
h5_batch, y_batch = hidden5_train[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={hidden5_out: h5_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={hidden5_out: hidden5_valid, y: y_valid2})
if loss_val < best_loss:
save_path = five_frozen_saver.save(sess, "./my_mnist_model_5_to_9_five_frozen")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
t1 = time.time()
print("Total training time: {:.1f}s".format(t1 - t0))
with tf.Session() as sess:
five_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_five_frozen")
acc_test = accuracy.eval(feed_dict={X: X_test2, y: y_test2})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
INFO:tensorflow:Restoring parameters from ./my_best_mnist_model_0_to_4 0 Validation loss: 1.029333 Best loss: 1.029333 Accuracy: 66.67% 1 Validation loss: 0.742783 Best loss: 0.742783 Accuracy: 73.33% 2 Validation loss: 1.038488 Best loss: 0.742783 Accuracy: 62.00% 3 Validation loss: 0.800629 Best loss: 0.742783 Accuracy: 71.33% 4 Validation loss: 0.910742 Best loss: 0.742783 Accuracy: 68.00% 5 Validation loss: 0.772327 Best loss: 0.742783 Accuracy: 71.33% 6 Validation loss: 0.785918 Best loss: 0.742783 Accuracy: 72.00% 7 Validation loss: 0.735777 Best loss: 0.735777 Accuracy: 74.67% 8 Validation loss: 0.722952 Best loss: 0.722952 Accuracy: 74.00% 9 Validation loss: 0.715727 Best loss: 0.715727 Accuracy: 76.67% 10 Validation loss: 0.763458 Best loss: 0.715727 Accuracy: 74.00% 11 Validation loss: 0.841518 Best loss: 0.715727 Accuracy: 71.33% 12 Validation loss: 0.712917 Best loss: 0.712917 Accuracy: 75.33% 13 Validation loss: 0.769865 Best loss: 0.712917 Accuracy: 72.00% 14 Validation loss: 0.870910 Best loss: 0.712917 Accuracy: 67.33% 15 Validation loss: 0.793758 Best loss: 0.712917 Accuracy: 72.00% 16 Validation loss: 0.737047 Best loss: 0.712917 Accuracy: 74.00% 17 Validation loss: 0.787298 Best loss: 0.712917 Accuracy: 74.00% 18 Validation loss: 0.814926 Best loss: 0.712917 Accuracy: 71.33% 19 Validation loss: 0.952072 Best loss: 0.712917 Accuracy: 67.33% 20 Validation loss: 0.754561 Best loss: 0.712917 Accuracy: 74.00% 21 Validation loss: 0.728781 Best loss: 0.712917 Accuracy: 73.33% 22 Validation loss: 0.730831 Best loss: 0.712917 Accuracy: 75.33% 23 Validation loss: 0.671014 Best loss: 0.671014 Accuracy: 76.00% 24 Validation loss: 0.686939 Best loss: 0.671014 Accuracy: 74.67% 25 Validation loss: 0.736364 Best loss: 0.671014 Accuracy: 74.67% 26 Validation loss: 0.720962 Best loss: 0.671014 Accuracy: 76.67% 27 Validation loss: 0.703039 Best loss: 0.671014 Accuracy: 78.00% 28 Validation loss: 0.688486 Best loss: 0.671014 Accuracy: 77.33% 29 Validation loss: 0.679878 Best loss: 0.671014 Accuracy: 74.67% 30 Validation loss: 0.705512 Best loss: 0.671014 Accuracy: 75.33% 31 Validation loss: 0.710309 Best loss: 0.671014 Accuracy: 78.00% 32 Validation loss: 0.754995 Best loss: 0.671014 Accuracy: 74.67% 33 Validation loss: 0.663437 Best loss: 0.663437 Accuracy: 78.67% 34 Validation loss: 0.816538 Best loss: 0.663437 Accuracy: 74.67% 35 Validation loss: 0.876961 Best loss: 0.663437 Accuracy: 70.00% 36 Validation loss: 0.918264 Best loss: 0.663437 Accuracy: 71.33% 37 Validation loss: 0.940113 Best loss: 0.663437 Accuracy: 70.00% 38 Validation loss: 0.788815 Best loss: 0.663437 Accuracy: 75.33% 39 Validation loss: 0.704214 Best loss: 0.663437 Accuracy: 76.00% 40 Validation loss: 0.724946 Best loss: 0.663437 Accuracy: 76.00% 41 Validation loss: 0.732040 Best loss: 0.663437 Accuracy: 75.33% 42 Validation loss: 0.798907 Best loss: 0.663437 Accuracy: 75.33% 43 Validation loss: 0.894106 Best loss: 0.663437 Accuracy: 72.67% 44 Validation loss: 0.699822 Best loss: 0.663437 Accuracy: 76.67% 45 Validation loss: 0.741509 Best loss: 0.663437 Accuracy: 76.67% 46 Validation loss: 0.723678 Best loss: 0.663437 Accuracy: 77.33% 47 Validation loss: 0.828757 Best loss: 0.663437 Accuracy: 70.67% 48 Validation loss: 0.802254 Best loss: 0.663437 Accuracy: 75.33% 49 Validation loss: 0.731748 Best loss: 0.663437 Accuracy: 76.67% 50 Validation loss: 0.811811 Best loss: 0.663437 Accuracy: 75.33% 51 Validation loss: 0.882759 Best loss: 0.663437 Accuracy: 73.33% 52 Validation loss: 1.058960 Best loss: 0.663437 Accuracy: 69.33% 53 Validation loss: 0.680124 Best loss: 0.663437 Accuracy: 74.67% Early stopping! Total training time: 2.2s INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_five_frozen Final test accuracy: 74.04%
9.4.¶
Exercise: try again reusing just four hidden layers instead of five. Can you achieve a higher precision?
Let's load the best model again, but this time we will create a new softmax output layer on top of the 4th hidden layer:
In [148]:
reset_graph()
n_outputs = 5
restore_saver = tf.train.import_meta_graph("./my_best_mnist_model_0_to_4.meta")
X = tf.get_default_graph().get_tensor_by_name("X:0")
y = tf.get_default_graph().get_tensor_by_name("y:0")
hidden4_out = tf.get_default_graph().get_tensor_by_name("hidden4_out:0")
logits = tf.layers.dense(hidden4_out, n_outputs, kernel_initializer=he_init, name="new_logits")
Y_proba = tf.nn.softmax(logits)
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name="accuracy")
And now let's create the training operation. We want to freeze all the layers except for the new output layer:
In [149]:
learning_rate = 0.01
output_layer_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope="new_logits")
optimizer = tf.train.AdamOptimizer(learning_rate, name="Adam2")
training_op = optimizer.minimize(loss, var_list=output_layer_vars)
init = tf.global_variables_initializer()
four_frozen_saver = tf.train.Saver()
And once again we train the model with the same code as earlier. Note: we could of course write a function once and use it multiple times, rather than copying almost the same training code over and over again, but as we keep tweaking the code slightly, the function would need multiple arguments and if statements, and it would have to be at the beginning of the notebook, where it would not make much sense to readers. In short it would be very confusing, so we're better off with copy & paste.
In [150]:
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_best_mnist_model_0_to_4")
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid2, y: y_valid2})
if loss_val < best_loss:
save_path = four_frozen_saver.save(sess, "./my_mnist_model_5_to_9_four_frozen")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
with tf.Session() as sess:
four_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_four_frozen")
acc_test = accuracy.eval(feed_dict={X: X_test2, y: y_test2})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
INFO:tensorflow:Restoring parameters from ./my_best_mnist_model_0_to_4 0 Validation loss: 0.881422 Best loss: 0.881422 Accuracy: 72.00% 1 Validation loss: 0.968628 Best loss: 0.881422 Accuracy: 72.67% 2 Validation loss: 0.750038 Best loss: 0.750038 Accuracy: 74.00% 3 Validation loss: 0.740100 Best loss: 0.740100 Accuracy: 74.00% 4 Validation loss: 0.754372 Best loss: 0.740100 Accuracy: 76.67% 5 Validation loss: 0.737881 Best loss: 0.737881 Accuracy: 72.67% 6 Validation loss: 0.734858 Best loss: 0.734858 Accuracy: 73.33% 7 Validation loss: 0.642497 Best loss: 0.642497 Accuracy: 80.00% 8 Validation loss: 0.675533 Best loss: 0.642497 Accuracy: 80.00% 9 Validation loss: 0.622721 Best loss: 0.622721 Accuracy: 80.00% 10 Validation loss: 0.758810 Best loss: 0.622721 Accuracy: 76.67% 11 Validation loss: 0.632438 Best loss: 0.622721 Accuracy: 78.00% 12 Validation loss: 0.651653 Best loss: 0.622721 Accuracy: 78.67% 13 Validation loss: 0.620286 Best loss: 0.620286 Accuracy: 78.67% 14 Validation loss: 0.721464 Best loss: 0.620286 Accuracy: 74.00% 15 Validation loss: 0.685866 Best loss: 0.620286 Accuracy: 78.00% 16 Validation loss: 0.691669 Best loss: 0.620286 Accuracy: 77.33% 17 Validation loss: 0.631125 Best loss: 0.620286 Accuracy: 78.67% 18 Validation loss: 0.627398 Best loss: 0.620286 Accuracy: 78.67% 19 Validation loss: 0.591787 Best loss: 0.591787 Accuracy: 81.33% 20 Validation loss: 0.600634 Best loss: 0.591787 Accuracy: 79.33% 21 Validation loss: 0.699794 Best loss: 0.591787 Accuracy: 77.33% 22 Validation loss: 0.564104 Best loss: 0.564104 Accuracy: 82.00% 23 Validation loss: 0.583253 Best loss: 0.564104 Accuracy: 81.33% 24 Validation loss: 0.647205 Best loss: 0.564104 Accuracy: 82.67% 25 Validation loss: 0.624429 Best loss: 0.564104 Accuracy: 82.67% 26 Validation loss: 0.672697 Best loss: 0.564104 Accuracy: 80.00% 27 Validation loss: 0.617655 Best loss: 0.564104 Accuracy: 80.00% 28 Validation loss: 0.604995 Best loss: 0.564104 Accuracy: 82.00% 29 Validation loss: 0.593213 Best loss: 0.564104 Accuracy: 80.67% 30 Validation loss: 0.613151 Best loss: 0.564104 Accuracy: 78.67% 31 Validation loss: 0.637576 Best loss: 0.564104 Accuracy: 82.00% 32 Validation loss: 0.595691 Best loss: 0.564104 Accuracy: 82.00% 33 Validation loss: 0.586561 Best loss: 0.564104 Accuracy: 80.67% 34 Validation loss: 0.633014 Best loss: 0.564104 Accuracy: 80.67% 35 Validation loss: 0.622603 Best loss: 0.564104 Accuracy: 82.00% 36 Validation loss: 0.657584 Best loss: 0.564104 Accuracy: 81.33% 37 Validation loss: 0.652834 Best loss: 0.564104 Accuracy: 78.00% 38 Validation loss: 0.620781 Best loss: 0.564104 Accuracy: 82.67% 39 Validation loss: 0.607198 Best loss: 0.564104 Accuracy: 81.33% 40 Validation loss: 0.587342 Best loss: 0.564104 Accuracy: 80.00% 41 Validation loss: 0.597247 Best loss: 0.564104 Accuracy: 82.67% 42 Validation loss: 0.588240 Best loss: 0.564104 Accuracy: 82.00% Early stopping! INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_four_frozen Final test accuracy: 75.77%
Still not fantastic, but much better.
9.5.¶
Exercise: now unfreeze the top two hidden layers and continue training: can you get the model to perform even better?
In [151]:
learning_rate = 0.01
unfrozen_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope="hidden[34]|new_logits")
optimizer = tf.train.AdamOptimizer(learning_rate, name="Adam3")
training_op = optimizer.minimize(loss, var_list=unfrozen_vars)
init = tf.global_variables_initializer()
two_frozen_saver = tf.train.Saver()
In [152]:
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
four_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_four_frozen")
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid2, y: y_valid2})
if loss_val < best_loss:
save_path = two_frozen_saver.save(sess, "./my_mnist_model_5_to_9_two_frozen")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
with tf.Session() as sess:
two_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_two_frozen")
acc_test = accuracy.eval(feed_dict={X: X_test2, y: y_test2})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_four_frozen 0 Validation loss: 1.696937 Best loss: 1.696937 Accuracy: 60.67% 1 Validation loss: 0.747527 Best loss: 0.747527 Accuracy: 80.67% 2 Validation loss: 0.905535 Best loss: 0.747527 Accuracy: 82.67% 3 Validation loss: 0.668797 Best loss: 0.668797 Accuracy: 83.33% 4 Validation loss: 0.610470 Best loss: 0.610470 Accuracy: 86.00% 5 Validation loss: 0.603745 Best loss: 0.603745 Accuracy: 86.67% 6 Validation loss: 0.983754 Best loss: 0.603745 Accuracy: 82.67% 7 Validation loss: 0.677826 Best loss: 0.603745 Accuracy: 82.00% 8 Validation loss: 0.747910 Best loss: 0.603745 Accuracy: 82.67% 9 Validation loss: 0.752873 Best loss: 0.603745 Accuracy: 84.00% 10 Validation loss: 0.874549 Best loss: 0.603745 Accuracy: 84.00% 11 Validation loss: 1.100320 Best loss: 0.603745 Accuracy: 74.00% 12 Validation loss: 0.850704 Best loss: 0.603745 Accuracy: 82.67% 13 Validation loss: 0.635135 Best loss: 0.603745 Accuracy: 82.00% 14 Validation loss: 1.022296 Best loss: 0.603745 Accuracy: 82.00% 15 Validation loss: 0.924854 Best loss: 0.603745 Accuracy: 84.00% 16 Validation loss: 0.585629 Best loss: 0.585629 Accuracy: 88.00% 17 Validation loss: 0.926777 Best loss: 0.585629 Accuracy: 82.67% 18 Validation loss: 0.681369 Best loss: 0.585629 Accuracy: 86.00% 19 Validation loss: 1.123767 Best loss: 0.585629 Accuracy: 83.33% 20 Validation loss: 0.677808 Best loss: 0.585629 Accuracy: 88.67% 21 Validation loss: 0.852799 Best loss: 0.585629 Accuracy: 84.67% 22 Validation loss: 0.948069 Best loss: 0.585629 Accuracy: 85.33% 23 Validation loss: 1.372686 Best loss: 0.585629 Accuracy: 84.00% 24 Validation loss: 1.537495 Best loss: 0.585629 Accuracy: 82.67% 25 Validation loss: 0.890086 Best loss: 0.585629 Accuracy: 86.67% 26 Validation loss: 0.881225 Best loss: 0.585629 Accuracy: 84.67% 27 Validation loss: 1.258317 Best loss: 0.585629 Accuracy: 83.33% 28 Validation loss: 2.230973 Best loss: 0.585629 Accuracy: 79.33% 29 Validation loss: 1.063308 Best loss: 0.585629 Accuracy: 84.00% 30 Validation loss: 1.082618 Best loss: 0.585629 Accuracy: 82.67% 31 Validation loss: 1.089194 Best loss: 0.585629 Accuracy: 84.00% 32 Validation loss: 1.658390 Best loss: 0.585629 Accuracy: 82.67% 33 Validation loss: 1.607493 Best loss: 0.585629 Accuracy: 82.67% 34 Validation loss: 1.052044 Best loss: 0.585629 Accuracy: 83.33% 35 Validation loss: 1.122056 Best loss: 0.585629 Accuracy: 86.00% 36 Validation loss: 1.243778 Best loss: 0.585629 Accuracy: 81.33% Early stopping! INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_two_frozen Final test accuracy: 85.81%
Let's check what accuracy we can get by unfreezing all layers:
In [153]:
learning_rate = 0.01
optimizer = tf.train.AdamOptimizer(learning_rate, name="Adam4")
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
no_frozen_saver = tf.train.Saver()
In [154]:
n_epochs = 1000
batch_size = 20
max_checks_without_progress = 20
checks_without_progress = 0
best_loss = np.infty
with tf.Session() as sess:
init.run()
two_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_two_frozen")
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
loss_val, acc_val = sess.run([loss, accuracy], feed_dict={X: X_valid2, y: y_valid2})
if loss_val < best_loss:
save_path = no_frozen_saver.save(sess, "./my_mnist_model_5_to_9_no_frozen")
best_loss = loss_val
checks_without_progress = 0
else:
checks_without_progress += 1
if checks_without_progress > max_checks_without_progress:
print("Early stopping!")
break
print("{}\tValidation loss: {:.6f}\tBest loss: {:.6f}\tAccuracy: {:.2f}%".format(
epoch, loss_val, best_loss, acc_val * 100))
with tf.Session() as sess:
no_frozen_saver.restore(sess, "./my_mnist_model_5_to_9_no_frozen")
acc_test = accuracy.eval(feed_dict={X: X_test2, y: y_test2})
print("Final test accuracy: {:.2f}%".format(acc_test * 100))
INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_two_frozen 0 Validation loss: 0.988528 Best loss: 0.988528 Accuracy: 68.67% 1 Validation loss: 1.030876 Best loss: 0.988528 Accuracy: 74.00% 2 Validation loss: 1.024854 Best loss: 0.988528 Accuracy: 90.67% 3 Validation loss: 1.109810 Best loss: 0.988528 Accuracy: 90.00% 4 Validation loss: 0.852205 Best loss: 0.852205 Accuracy: 86.67% 5 Validation loss: 1.024187 Best loss: 0.852205 Accuracy: 88.67% 6 Validation loss: 1.784481 Best loss: 0.852205 Accuracy: 88.00% 7 Validation loss: 1.737455 Best loss: 0.852205 Accuracy: 82.67% 8 Validation loss: 1.204837 Best loss: 0.852205 Accuracy: 90.00% 9 Validation loss: 1.421492 Best loss: 0.852205 Accuracy: 90.00% 10 Validation loss: 2.262671 Best loss: 0.852205 Accuracy: 90.00% 11 Validation loss: 2.174999 Best loss: 0.852205 Accuracy: 88.00% 12 Validation loss: 4.743779 Best loss: 0.852205 Accuracy: 80.00% 13 Validation loss: 2.013788 Best loss: 0.852205 Accuracy: 88.00% 14 Validation loss: 3.007850 Best loss: 0.852205 Accuracy: 90.00% 15 Validation loss: 1.136518 Best loss: 0.852205 Accuracy: 88.67% 16 Validation loss: 1.765155 Best loss: 0.852205 Accuracy: 86.67% 17 Validation loss: 7.687177 Best loss: 0.852205 Accuracy: 83.33% 18 Validation loss: 5.636858 Best loss: 0.852205 Accuracy: 55.33% 19 Validation loss: 40.321644 Best loss: 0.852205 Accuracy: 69.33% 20 Validation loss: 18.057196 Best loss: 0.852205 Accuracy: 80.67% 21 Validation loss: 0.830525 Best loss: 0.830525 Accuracy: 81.33% 22 Validation loss: 10.426401 Best loss: 0.830525 Accuracy: 78.67% 23 Validation loss: 21.855406 Best loss: 0.830525 Accuracy: 85.33% 24 Validation loss: 32.382896 Best loss: 0.830525 Accuracy: 88.00% 25 Validation loss: 37.215786 Best loss: 0.830525 Accuracy: 85.33% 26 Validation loss: 41.230343 Best loss: 0.830525 Accuracy: 90.67% 27 Validation loss: 44.421844 Best loss: 0.830525 Accuracy: 88.00% 28 Validation loss: 55.242134 Best loss: 0.830525 Accuracy: 90.00% 29 Validation loss: 55.446503 Best loss: 0.830525 Accuracy: 92.00% 30 Validation loss: 55.016945 Best loss: 0.830525 Accuracy: 92.67% 31 Validation loss: 56.274517 Best loss: 0.830525 Accuracy: 92.67% 32 Validation loss: 56.045486 Best loss: 0.830525 Accuracy: 92.67% 33 Validation loss: 55.850620 Best loss: 0.830525 Accuracy: 92.67% 34 Validation loss: 55.911472 Best loss: 0.830525 Accuracy: 92.67% 35 Validation loss: 55.964947 Best loss: 0.830525 Accuracy: 92.67% 36 Validation loss: 55.996971 Best loss: 0.830525 Accuracy: 92.67% 37 Validation loss: 56.012630 Best loss: 0.830525 Accuracy: 92.67% 38 Validation loss: 56.026680 Best loss: 0.830525 Accuracy: 92.67% 39 Validation loss: 56.026505 Best loss: 0.830525 Accuracy: 92.67% 40 Validation loss: 56.046001 Best loss: 0.830525 Accuracy: 92.67% 41 Validation loss: 56.039871 Best loss: 0.830525 Accuracy: 92.67% Early stopping! INFO:tensorflow:Restoring parameters from ./my_mnist_model_5_to_9_no_frozen Final test accuracy: 78.54%
Let's compare that to a DNN trained from scratch:
In [155]:
dnn_clf_5_to_9 = DNNClassifier(n_hidden_layers=4, random_state=42)
dnn_clf_5_to_9.fit(X_train2, y_train2, n_epochs=1000, X_valid=X_valid2, y_valid=y_valid2)
0 Validation loss: 0.637013 Best loss: 0.637013 Accuracy: 80.67% 1 Validation loss: 0.618201 Best loss: 0.618201 Accuracy: 87.33% 2 Validation loss: 1.831457 Best loss: 0.618201 Accuracy: 75.33% 3 Validation loss: 0.661379 Best loss: 0.618201 Accuracy: 87.33% 4 Validation loss: 0.568603 Best loss: 0.568603 Accuracy: 86.67% 5 Validation loss: 0.753698 Best loss: 0.568603 Accuracy: 91.33% 6 Validation loss: 1.085846 Best loss: 0.568603 Accuracy: 86.00% 7 Validation loss: 0.616958 Best loss: 0.568603 Accuracy: 93.33% 8 Validation loss: 0.920785 Best loss: 0.568603 Accuracy: 85.33% 9 Validation loss: 1.262947 Best loss: 0.568603 Accuracy: 91.33% 10 Validation loss: 0.972570 Best loss: 0.568603 Accuracy: 89.33% 11 Validation loss: 1.382815 Best loss: 0.568603 Accuracy: 85.33% 12 Validation loss: 1.854567 Best loss: 0.568603 Accuracy: 87.33% 13 Validation loss: 1.598570 Best loss: 0.568603 Accuracy: 88.67% 14 Validation loss: 1.567680 Best loss: 0.568603 Accuracy: 91.33% 15 Validation loss: 1.731191 Best loss: 0.568603 Accuracy: 89.33% 16 Validation loss: 1.939850 Best loss: 0.568603 Accuracy: 90.67% 17 Validation loss: 1.931630 Best loss: 0.568603 Accuracy: 88.00% 18 Validation loss: 1.972910 Best loss: 0.568603 Accuracy: 88.00% 19 Validation loss: 1.223359 Best loss: 0.568603 Accuracy: 90.67% 20 Validation loss: 1.165815 Best loss: 0.568603 Accuracy: 88.67% 21 Validation loss: 1.374972 Best loss: 0.568603 Accuracy: 91.33% 22 Validation loss: 1.450595 Best loss: 0.568603 Accuracy: 91.33% 23 Validation loss: 1.465558 Best loss: 0.568603 Accuracy: 91.33% 24 Validation loss: 1.471532 Best loss: 0.568603 Accuracy: 91.33% 25 Validation loss: 1.475235 Best loss: 0.568603 Accuracy: 91.33% Early stopping!
Out[155]:
DNNClassifier(activation=<function elu at 0x7f95d738dbf8>,
batch_norm_momentum=None, batch_size=20, dropout_rate=None,
initializer=<tensorflow.python.ops.init_ops.VarianceScaling object at 0x7f94ce9dd390>,
learning_rate=0.01, n_hidden_layers=4, n_neurons=100,
optimizer_class=<class 'tensorflow.python.training.adam.AdamOptimizer'>,
random_state=42)
In [156]:
y_pred = dnn_clf_5_to_9.predict(X_test2)
accuracy_score(y_test2, y_pred)
Out[156]:
0.8718370705616129
Meh. How disappointing! ;) Transfer learning did not help much (if at all) in this task. At least we tried... Fortunately, the next exercise will get better results.
10. Pretraining on an auxiliary task¶
In this exercise you will build a DNN that compares two MNIST digit images and predicts whether they represent the same digit or not. Then you will reuse the lower layers of this network to train an MNIST classifier using very little training data.
10.1.¶
Exercise: Start by building two DNNs (let's call them DNN A and B), both similar to the one you built earlier but without the output layer: each DNN should have five hidden layers of 100 neurons each, He initialization, and ELU activation. Next, add one more hidden layer with 10 units on top of both DNNs. You should use TensorFlow's concat() function with axis=1 to concatenate the outputs of both DNNs along the horizontal axis, then feed the result to the hidden layer. Finally, add an output layer with a single neuron using the logistic activation function.
Warning! There was an error in the book for this exercise: there was no instruction to add a top hidden layer. Without it, the neural network generally fails to start learning. If you have the latest version of the book, this error has been fixed.
You could have two input placeholders, X1 and X2, one for the images that should be fed to the first DNN, and the other for the images that should be fed to the second DNN. It would work fine. However, another option is to have a single input placeholder to hold both sets of images (each row will hold a pair of images), and use tf.unstack() to split this tensor into two separate tensors, like this:
In [157]:
n_inputs = 28 * 28 # MNIST
reset_graph()
X = tf.placeholder(tf.float32, shape=(None, 2, n_inputs), name="X")
X1, X2 = tf.unstack(X, axis=1)
We also need the labels placeholder. Each label will be 0 if the images represent different digits, or 1 if they represent the same digit:
In [158]:
y = tf.placeholder(tf.int32, shape=[None, 1])
Now let's feed these inputs through two separate DNNs:
In [159]:
dnn1 = dnn(X1, name="DNN_A")
dnn2 = dnn(X2, name="DNN_B")
And let's concatenate their outputs:
In [160]:
dnn_outputs = tf.concat([dnn1, dnn2], axis=1)
Each DNN outputs 100 activations (per instance), so the shape is [None, 100]:
In [161]:
dnn1.shape
Out[161]:
TensorShape([Dimension(None), Dimension(100)])
In [162]:
dnn2.shape
Out[162]:
TensorShape([Dimension(None), Dimension(100)])
And of course the concatenated outputs have a shape of [None, 200]:
In [163]:
dnn_outputs.shape
Out[163]:
TensorShape([Dimension(None), Dimension(200)])
Now lets add an extra hidden layer with just 10 neurons, and the output layer, with a single neuron:
In [164]:
hidden = tf.layers.dense(dnn_outputs, units=10, activation=tf.nn.elu, kernel_initializer=he_init)
logits = tf.layers.dense(hidden, units=1, kernel_initializer=he_init)
y_proba = tf.nn.sigmoid(logits)
The whole network predicts 1 if y_proba >= 0.5 (i.e. the network predicts that the images represent the same digit), or 0 otherwise. We compute instead logits >= 0, which is equivalent but faster to compute:
In [165]:
y_pred = tf.cast(tf.greater_equal(logits, 0), tf.int32)
Now let's add the cost function:
In [166]:
y_as_float = tf.cast(y, tf.float32)
xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_as_float, logits=logits)
loss = tf.reduce_mean(xentropy)
And we can now create the training operation using an optimizer:
In [167]:
learning_rate = 0.01
momentum = 0.95
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum, use_nesterov=True)
training_op = optimizer.minimize(loss)
We will want to measure our classifier's accuracy.
In [168]:
y_pred_correct = tf.equal(y_pred, y)
accuracy = tf.reduce_mean(tf.cast(y_pred_correct, tf.float32))
And the usual init and saver:
In [169]:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
10.2.¶
Exercise: split the MNIST training set in two sets: split #1 should containing 55,000 images, and split #2 should contain contain 5,000 images. Create a function that generates a training batch where each instance is a pair of MNIST images picked from split #1. Half of the training instances should be pairs of images that belong to the same class, while the other half should be images from different classes. For each pair, the training label should be 0 if the images are from the same class, or 1 if they are from different classes.
The MNIST dataset returned by TensorFlow's input_data() function is already split into 3 parts: a training set (55,000 instances), a validation set (5,000 instances) and a test set (10,000 instances). Let's use the first set to generate the training set composed image pairs, and we will use the second set for the second phase of the exercise (to train a regular MNIST classifier). We will use the third set as the test set for both phases.
In [170]:
X_train1 = X_train
y_train1 = y_train
X_train2 = X_valid
y_train2 = y_valid
X_test = X_test
y_test = y_test
Let's write a function that generates pairs of images: 50% representing the same digit, and 50% representing different digits. There are many ways to implement this. In this implementation, we first decide how many "same" pairs (i.e. pairs of images representing the same digit) we will generate, and how many "different" pairs (i.e. pairs of images representing different digits). We could just use batch_size // 2 but we want to handle the case where it is odd (granted, that might be overkill!). Then we generate random pairs and we pick the right number of "same" pairs, then we generate the right number of "different" pairs. Finally we shuffle the batch and return it:
In [171]:
def generate_batch(images, labels, batch_size):
size1 = batch_size // 2
size2 = batch_size - size1
if size1 != size2 and np.random.rand() > 0.5:
size1, size2 = size2, size1
X = []
y = []
while len(X) < size1:
rnd_idx1, rnd_idx2 = np.random.randint(0, len(images), 2)
if rnd_idx1 != rnd_idx2 and labels[rnd_idx1] == labels[rnd_idx2]:
X.append(np.array([images[rnd_idx1], images[rnd_idx2]]))
y.append([1])
while len(X) < batch_size:
rnd_idx1, rnd_idx2 = np.random.randint(0, len(images), 2)
if labels[rnd_idx1] != labels[rnd_idx2]:
X.append(np.array([images[rnd_idx1], images[rnd_idx2]]))
y.append([0])
rnd_indices = np.random.permutation(batch_size)
return np.array(X)[rnd_indices], np.array(y)[rnd_indices]
Let's test it to generate a small batch of 5 image pairs:
In [172]:
batch_size = 5
X_batch, y_batch = generate_batch(X_train1, y_train1, batch_size)
Each row in X_batch contains a pair of images:
In [173]:
X_batch.shape, X_batch.dtype
Out[173]:
((5, 2, 784), dtype('float32'))
Let's look at these pairs:
In [174]:
plt.figure(figsize=(3, 3 * batch_size))
plt.subplot(121)
plt.imshow(X_batch[:,0].reshape(28 * batch_size, 28), cmap="binary", interpolation="nearest")
plt.axis('off')
plt.subplot(122)
plt.imshow(X_batch[:,1].reshape(28 * batch_size, 28), cmap="binary", interpolation="nearest")
plt.axis('off')
plt.show()
And let's look at the labels (0 means "different", 1 means "same"):
In [175]:
y_batch
Out[175]:
array([[1],
[0],
[0],
[1],
[0]])
Perfect!
10.3.¶
Exercise: train the DNN on this training set. For each image pair, you can simultaneously feed the first image to DNN A and the second image to DNN B. The whole network will gradually learn to tell whether two images belong to the same class or not.
Let's generate a test set composed of many pairs of images pulled from the MNIST test set:
In [176]:
X_test1, y_test1 = generate_batch(X_test, y_test, batch_size=len(X_test))
And now, let's train the model. There's really nothing special about this step, except for the fact that we need a fairly large batch_size, otherwise the model fails to learn anything and ends up with an accuracy of 50%:
In [177]:
n_epochs = 100
batch_size = 500
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
for iteration in range(len(X_train1) // batch_size):
X_batch, y_batch = generate_batch(X_train1, y_train1, batch_size)
loss_val, _ = sess.run([loss, training_op], feed_dict={X: X_batch, y: y_batch})
print(epoch, "Train loss:", loss_val)
if epoch % 5 == 0:
acc_test = accuracy.eval(feed_dict={X: X_test1, y: y_test1})
print(epoch, "Test accuracy:", acc_test)
save_path = saver.save(sess, "./my_digit_comparison_model.ckpt")
0 Train loss: 0.693449 0 Test accuracy: 0.5091 1 Train loss: 0.6925517 2 Train loss: 0.6913698 3 Train loss: 0.69394815 4 Train loss: 0.69199574 5 Train loss: 0.6912017 5 Test accuracy: 0.4997 6 Train loss: 0.69237447 7 Train loss: 0.67959845 8 Train loss: 0.58347875 9 Train loss: 0.5501087 10 Train loss: 0.47479486 10 Test accuracy: 0.747 11 Train loss: 0.47786787 12 Train loss: 0.43815765 13 Train loss: 0.4584329 14 Train loss: 0.37488955 15 Train loss: 0.3944795 15 Test accuracy: 0.8408 16 Train loss: 0.35263225 17 Train loss: 0.3779824 18 Train loss: 0.31390512 19 Train loss: 0.34035394 20 Train loss: 0.2987433 20 Test accuracy: 0.8921 21 Train loss: 0.23179962 22 Train loss: 0.23321524 23 Train loss: 0.22762412 24 Train loss: 0.21296309 25 Train loss: 0.19437297 25 Test accuracy: 0.915 26 Train loss: 0.22710522 27 Train loss: 0.19624695 28 Train loss: 0.20233664 29 Train loss: 0.16502927 30 Train loss: 0.19200236 30 Test accuracy: 0.9265 31 Train loss: 0.18807735 32 Train loss: 0.15258598 33 Train loss: 0.11344788 34 Train loss: 0.20222928 35 Train loss: 0.13183561 35 Test accuracy: 0.9371 36 Train loss: 0.12587659 37 Train loss: 0.07443967 38 Train loss: 0.09581424 39 Train loss: 0.13854165 40 Train loss: 0.13258658 40 Test accuracy: 0.949 41 Train loss: 0.15852296 42 Train loss: 0.11823679 43 Train loss: 0.13090087 44 Train loss: 0.121523395 45 Train loss: 0.21524942 45 Test accuracy: 0.9537 46 Train loss: 0.093182944 47 Train loss: 0.10293961 48 Train loss: 0.080560714 49 Train loss: 0.067396514 50 Train loss: 0.07696808 50 Test accuracy: 0.9569 51 Train loss: 0.09675123 52 Train loss: 0.07796277 53 Train loss: 0.13375118 54 Train loss: 0.10105775 55 Train loss: 0.08282135 55 Test accuracy: 0.9619 56 Train loss: 0.07425493 57 Train loss: 0.06667148 58 Train loss: 0.09082972 59 Train loss: 0.10377218 60 Train loss: 0.08135713 60 Test accuracy: 0.9652 61 Train loss: 0.058796614 62 Train loss: 0.07295383 63 Train loss: 0.08076497 64 Train loss: 0.079403915 65 Train loss: 0.028367642 65 Test accuracy: 0.966 66 Train loss: 0.0645599 67 Train loss: 0.05789566 68 Train loss: 0.055227682 69 Train loss: 0.05531174 70 Train loss: 0.08921583 70 Test accuracy: 0.9668 71 Train loss: 0.041489303 72 Train loss: 0.058670394 73 Train loss: 0.058065996 74 Train loss: 0.081077345 75 Train loss: 0.0547798 75 Test accuracy: 0.969 76 Train loss: 0.03891569 77 Train loss: 0.0462587 78 Train loss: 0.051314954 79 Train loss: 0.047135957 80 Train loss: 0.025371503 80 Test accuracy: 0.9693 81 Train loss: 0.049629647 82 Train loss: 0.026197875 83 Train loss: 0.057823546 84 Train loss: 0.025820913 85 Train loss: 0.043315757 85 Test accuracy: 0.9717 86 Train loss: 0.031463966 87 Train loss: 0.05097594 88 Train loss: 0.047270138 89 Train loss: 0.022847315 90 Train loss: 0.021965368 90 Test accuracy: 0.9719 91 Train loss: 0.03174819 92 Train loss: 0.042883325 93 Train loss: 0.030013515 94 Train loss: 0.035777166 95 Train loss: 0.033001903 95 Test accuracy: 0.9713 96 Train loss: 0.021019176 97 Train loss: 0.03824938 98 Train loss: 0.035014983 99 Train loss: 0.02618454
All right, we reach 97.6% accuracy on this digit comparison task. That's not too bad, this model knows a thing or two about comparing handwritten digits!
Let's see if some of that knowledge can be useful for the regular MNIST classification task.
10.4.¶
Exercise: now create a new DNN by reusing and freezing the hidden layers of DNN A and adding a softmax output layer on top with 10 neurons. Train this network on split #2 and see if you can achieve high performance despite having only 500 images per class.
Let's create the model, it is pretty straightforward. There are many ways to freeze the lower layers, as explained in the book. In this example, we chose to use the tf.stop_gradient() function. Note that we need one Saver to restore the pretrained DNN A, and another Saver to save the final model:
In [178]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
dnn_outputs = dnn(X, name="DNN_A")
frozen_outputs = tf.stop_gradient(dnn_outputs)
logits = tf.layers.dense(dnn_outputs, n_outputs, kernel_initializer=he_init)
Y_proba = tf.nn.softmax(logits)
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum, use_nesterov=True)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
dnn_A_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope="DNN_A")
restore_saver = tf.train.Saver(var_list={var.op.name: var for var in dnn_A_vars})
saver = tf.train.Saver()
Now on to training! We first initialize all variables (including the variables in the new output layer), then we restore the pretrained DNN A. Next, we just train the model on the small MNIST dataset (containing just 5,000 images):
In [179]:
n_epochs = 100
batch_size = 50
with tf.Session() as sess:
init.run()
restore_saver.restore(sess, "./my_digit_comparison_model.ckpt")
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
if epoch % 10 == 0:
acc_test = accuracy.eval(feed_dict={X: X_test, y: y_test})
print(epoch, "Test accuracy:", acc_test)
save_path = saver.save(sess, "./my_mnist_model_final.ckpt")
INFO:tensorflow:Restoring parameters from ./my_digit_comparison_model.ckpt 0 Test accuracy: 0.9418 10 Test accuracy: 0.96 20 Test accuracy: 0.9644 30 Test accuracy: 0.9641 40 Test accuracy: 0.9636 50 Test accuracy: 0.9636 60 Test accuracy: 0.9636 70 Test accuracy: 0.9636 80 Test accuracy: 0.9635 90 Test accuracy: 0.9634
Well, 96.7% accuracy, that's not the best MNIST model we have trained so far, but recall that we are only using a small training set (just 500 images per digit). Let's compare this result with the same DNN trained from scratch, without using transfer learning:
In [180]:
reset_graph()
n_inputs = 28 * 28 # MNIST
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int32, shape=(None), name="y")
dnn_outputs = dnn(X, name="DNN_A")
logits = tf.layers.dense(dnn_outputs, n_outputs, kernel_initializer=he_init)
Y_proba = tf.nn.softmax(logits)
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum, use_nesterov=True)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
dnn_A_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope="DNN_A")
restore_saver = tf.train.Saver(var_list={var.op.name: var for var in dnn_A_vars})
saver = tf.train.Saver()
In [181]:
n_epochs = 150
batch_size = 50
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
rnd_idx = np.random.permutation(len(X_train2))
for rnd_indices in np.array_split(rnd_idx, len(X_train2) // batch_size):
X_batch, y_batch = X_train2[rnd_indices], y_train2[rnd_indices]
sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
if epoch % 10 == 0:
acc_test = accuracy.eval(feed_dict={X: X_test, y: y_test})
print(epoch, "Test accuracy:", acc_test)
save_path = saver.save(sess, "./my_mnist_model_final.ckpt")
0 Test accuracy: 0.8611 10 Test accuracy: 0.9203 20 Test accuracy: 0.9242 30 Test accuracy: 0.9258 40 Test accuracy: 0.9426 50 Test accuracy: 0.9436 60 Test accuracy: 0.9442 70 Test accuracy: 0.9447 80 Test accuracy: 0.9449 90 Test accuracy: 0.9448 100 Test accuracy: 0.9445 110 Test accuracy: 0.9443 120 Test accuracy: 0.9443 130 Test accuracy: 0.9443 140 Test accuracy: 0.9443
Only 94.8% accuracy... So transfer learning helped us reduce the error rate from 5.2% to 3.3% (that's over 36% error reduction). Moreover, the model using transfer learning reached over 96% accuracy in less than 10 epochs.
Bottom line: transfer learning does not always work (as we saw in exercise 9), but when it does it can make a big difference. So try it out!
In [ ]: