Train#
The network can be trained using three interfaces:
python, via
das.train.trainthe command-line interface
das train.the GUI - see the GUI tutorial
Training will:
load train/val/test data form a dataset
initialize the network
save all parameters for reproducibility
train the network and save the best network to disk
run inference and evaluate the network using the test data.
The names of files created during training start with an optional prefix and the time stamp of the start time of training, as in my-awesome-prefix_20192310_091032. Typically, three files are created:
*_params.yaml- training parameters etc.*_model.h5- model architecture and weights*_results.h5- predictions and evaluation results for the test set (only created if the training dataset contains a test set)
Training using python#
Training is done using the train function in the das.train module:
import das.train
help(das.train.train)
Help on function train in module das.train:
train(*, data_dir: str, y_suffix: str = '', save_dir: str = './', save_prefix: Union[str, NoneType] = None, model_name: str = 'tcn', nb_filters: int = 16, kernel_size: int = 16, nb_conv: int = 3, use_separable: List[bool] = False, nb_hist: int = 1024, ignore_boundaries: bool = True, batch_norm: bool = True, nb_pre_conv: int = 0, pre_nb_dft: int = 64, pre_kernel_size: int = 3, pre_nb_filters: int = 16, pre_nb_conv: int = 2, nb_lstm_units: int = 0, verbose: int = 2, batch_size: int = 32, nb_epoch: int = 400, learning_rate: Union[float, NoneType] = None, reduce_lr: bool = False, reduce_lr_patience: int = 5, fraction_data: Union[float, NoneType] = None, seed: Union[int, NoneType] = None, batch_level_subsampling: bool = False, tensorboard: bool = False, neptune_api_token: Union[str, NoneType] = None, neptune_project: Union[str, NoneType] = None, log_messages: bool = False, nb_stacks: int = 2, with_y_hist: bool = True, x_suffix: str = '', balance: bool = False, version_data: bool = True, _qt_progress: bool = False) -> Tuple[keras.engine.training.Model, Dict[str, Any]]
Train a DeepSS network.
Args:
data_dir (str): Path to the directory or file with the dataset for training.
Accepts npy-dirs (recommended), h5 files or zarr files.
See documentation for how the dataset should be organized.
y_suffix (str): Select training target by suffix.
Song-type specific targets can be created with a training dataset,
Defaults to '' (will use the standard target 'y')
save_dir (str): Directory to save training outputs.
The path of output files will constructed from the SAVE_DIR, an optional prefix, and the time stamp of the start of training.
Defaults to current directory ('./').
save_prefix (Optional[str]): Prepend to timestamp.
Name of files created will be SAVE_DIR/SAVE_PREFIX + "_" + TIMESTAMP
or SAVE_DIR/ TIMESTAMP if SAVE_PREFIX is empty.
Defaults to '' (empty).
model_name (str): Network architecture to use.
Use "tcn" (TCN) or "tcn_stft" (TCN with STFT frontend).
See das.models for a description of all models.
Defaults to 'tcn'.
nb_filters (int): Number of filters per layer.
Defaults to 16.
kernel_size (int): Duration of the filters (=kernels) in samples.
Defaults to 16.
nb_conv (int): Number of TCN blocks in the network.
Defaults to 3.
use_separable (List[bool]): Specify which TCN blocks should use separable convolutions.
Provide as a space-separated sequence of "False" or "True.
For instance: "True False False" will set the first block in a
three-block (as given by nb_conv) network to use separable convolutions.
Defaults to False (no block uses separable convolution).
nb_hist (int): Number of samples processed at once by the network (a.k.a chunk size).
Defaults to 1024.
ignore_boundaries (bool): Minimize edge effects by discarding predictions at the edges of chunks.
Defaults to True.
batch_norm (bool): Batch normalize.
Defaults to True.
nb_pre_conv (int): Downsampling rate. Adds downsampling frontend if not 0.
TCN_TCN: adds a frontend of N conv blocks (conv-relu-batchnorm-maxpool2) to the TCN.
TCN_STFT: adds a trainable STFT frontend.
Defaults to 0 (no frontend).
pre_nb_dft (int): Number of filters (roughly corresponding to filters) in the STFT frontend.
Defaults to 64.
pre_nb_filters (int): Number of filters per layer in the pre-processing TCN.
Defaults to 16.
pre_kernel_size (int): Duration of filters (=kernels) in samples in the pre-processing TCN.
Defaults to 3.
nb_lstm_units (int): If >0, adds LSTM with given number of units to the output of the stack of TCN blocks.
Defaults to 0 (no LSTM layer).
verbose (int): Verbosity of training output (0 - no output(?), 1 - progress bar, 2 - one line per epoch).
Defaults to 2.
batch_size (int): Batch size
Defaults to 32.
nb_epoch (int): Maximal number of training epochs.
Training will stop early if validation loss did not decrease in the last 20 epochs.
Defaults to 400.
learning_rate (Optional[float]): Learning rate of the model. Defaults should work in most cases.
Values typically range between 0.1 and 0.00001.
If None, uses per model defaults: "tcn" 0.0001, "tcn_stft" 0.0005).
Defaults to None.
reduce_lr (bool): Reduce learning rate on plateau.
Defaults to False.
reduce_lr_patience (int): Number of epochs w/o a reduction in validation loss after which to trigger a reduction in learning rate.
Defaults to 5.
fraction_data (Optional[float]): Fraction of training and validation to use for training.
Defaults to 1.0.
seed (Optional[int]): Random seed to reproducible select fractions of the data.
Defaults to None (no seed).
batch_level_subsampling (bool): Select fraction of data for training from random subset of shuffled batches.
If False, select a continuous chunk of the recording.
Defaults to False.
tensorboard (bool): Write tensorboard logs to save_dir. Defaults to False.
neptune_api_token (Optional[str]): API token for logging to neptune.ai. Defaults to None (no logging).
neptune_project (Optional[str]): Project to log to for neptune.ai. Defaults to None (no logging).
log_messages (bool): Sets logging level to INFO.
Defaults to False (will follow existing settings).
nb_stacks (int): Unused if model name is "tcn" or "tcn_stft". Defaults to 2.
with_y_hist (bool): Unused if model name is "tcn" or "tcn_stft". Defaults to True.
x_suffix (str): Select specific training data based on suffix (e.g. x_suffix).
Defaults to '' (will use the standard data 'x')
balance (bool): Balance data. Weights class-wise errors by the inverse of the class frequencies.
Defaults to False.
version_data (bool): Save MD5 hash of the data_dir to log and params.yaml.
Defaults to True (set to False for large datasets since it can be slow).
Returns
model (keras.Model)
params (Dict[str, Any])
Calling the train function produce fairly verbose logging messages, to help with troubleshooting:
run time parameters
information on the size of the training and validation data
network architecture
training progress (training and validation loss)
after training, a classification report for the test data (if test data exist in the dataset)
When done, train returns the trained keras model and a parameter dictionary with all arguments required to reproduce the model.
To demonstrate the outputs of train, the following trains a small network on a small dataset to annotate pulse and sine song from Drosophila melanogater. Expected performance (f1-score) is about 75%.
model, params = das.train.train(model_name='tcn', # see `das.models` for valid model_names
data_dir='tutorial_dataset.npy',
save_dir='res',
nb_hist=256,
kernel_size=16,
nb_filters=16,
ignore_boundaries=True,
verbose=1,
nb_epoch=4,
log_messages=True)
INFO:root:Loading data from tutorial_dataset.npy.
INFO:root:Version of the data:
INFO:root: MD5 hash of tutorial_dataset.npy is
INFO:root: 34876fb30412a444e444a8e1f5312126
INFO:root:Parameters:
INFO:root:{'data_dir': 'tutorial_dataset.npy', 'y_suffix': '', 'save_dir': 'res', 'save_prefix': '', 'model_name': 'tcn', 'nb_filters': 16, 'kernel_size': 16, 'nb_conv': 3, 'use_separable': False, 'nb_hist': 256, 'ignore_boundaries': True, 'batch_norm': True, 'nb_pre_conv': 0, 'pre_nb_dft': 64, 'pre_kernel_size': 3, 'pre_nb_filters': 16, 'pre_nb_conv': 2, 'nb_lstm_units': 0, 'verbose': 1, 'batch_size': 32, 'nb_epoch': 4, 'reduce_lr': False, 'reduce_lr_patience': 5, 'fraction_data': None, 'seed': None, 'batch_level_subsampling': False, 'tensorboard': False, 'neptune_api_token': None, 'neptune_project': None, 'log_messages': True, 'nb_stacks': 2, 'with_y_hist': True, 'x_suffix': '', 'balance': False, 'version_data': True, 'sample_weight_mode': 'temporal', 'data_padding': 48, 'return_sequences': True, 'stride': 160, 'y_offset': 0, 'output_stride': 1, 'class_names': ['noise', 'pulse', 'sine'], 'class_names_pulse': ['noise', 'pulse'], 'class_names_sine': ['noise', 'sine'], 'class_types': ['segment', 'event', 'segment'], 'class_types_pulse': ['segment', 'event'], 'class_types_sine': ['segment', 'segment'], 'filename_endsample_test': [], 'filename_endsample_train': [], 'filename_endsample_val': [], 'filename_startsample_test': [], 'filename_startsample_train': [], 'filename_startsample_val': [], 'filename_train': [], 'filename_val': [], 'samplerate_x_Hz': 10000, 'samplerate_y_Hz': 10000, 'filename_test': [], 'data_hash': '34876fb30412a444e444a8e1f5312126', 'nb_freq': 1, 'nb_channels': 1, 'nb_classes': 3, 'first_sample_train': 0, 'last_sample_train': None, 'first_sample_val': 0, 'last_sample_val': None}
INFO:root:Preparing data
INFO:root:Training data:
INFO:root: AudioSequence with 3992 batches each with 32 items.
Total of 20440005 samples with
each x=(1,) and
each y=(3,)
INFO:root:Validation data:
INFO:root: AudioSequence with 812 batches each with 32 items.
Total of 4160001 samples with
each x=(1,) and
each y=(3,)
INFO:root:building network
/Users/janc/miniconda3/lib/python3.8/site-packages/keras/optimizer_v2/optimizer_v2.py:355: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
warnings.warn(
INFO:root:None
INFO:root:Will save to res/20210924_220702.
INFO:root:start training
Model: "TCN"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 256, 1)] 0
__________________________________________________________________________________________________
conv1d (Conv1D) (None, 256, 16) 32 input_1[0][0]
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, 256, 16) 4112 conv1d[0][0]
__________________________________________________________________________________________________
activation (Activation) (None, 256, 16) 0 conv1d_1[0][0]
__________________________________________________________________________________________________
lambda (Lambda) (None, 256, 16) 0 activation[0][0]
__________________________________________________________________________________________________
spatial_dropout1d (SpatialDropo (None, 256, 16) 0 lambda[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, 256, 16) 272 spatial_dropout1d[0][0]
__________________________________________________________________________________________________
add (Add) (None, 256, 16) 0 conv1d[0][0]
conv1d_2[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, 256, 16) 4112 add[0][0]
__________________________________________________________________________________________________
activation_1 (Activation) (None, 256, 16) 0 conv1d_3[0][0]
__________________________________________________________________________________________________
lambda_1 (Lambda) (None, 256, 16) 0 activation_1[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_1 (SpatialDro (None, 256, 16) 0 lambda_1[0][0]
__________________________________________________________________________________________________
conv1d_4 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_1[0][0]
__________________________________________________________________________________________________
add_1 (Add) (None, 256, 16) 0 add[0][0]
conv1d_4[0][0]
__________________________________________________________________________________________________
conv1d_5 (Conv1D) (None, 256, 16) 4112 add_1[0][0]
__________________________________________________________________________________________________
activation_2 (Activation) (None, 256, 16) 0 conv1d_5[0][0]
__________________________________________________________________________________________________
lambda_2 (Lambda) (None, 256, 16) 0 activation_2[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_2 (SpatialDro (None, 256, 16) 0 lambda_2[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_2[0][0]
__________________________________________________________________________________________________
add_2 (Add) (None, 256, 16) 0 add_1[0][0]
conv1d_6[0][0]
__________________________________________________________________________________________________
conv1d_7 (Conv1D) (None, 256, 16) 4112 add_2[0][0]
__________________________________________________________________________________________________
activation_3 (Activation) (None, 256, 16) 0 conv1d_7[0][0]
__________________________________________________________________________________________________
lambda_3 (Lambda) (None, 256, 16) 0 activation_3[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_3 (SpatialDro (None, 256, 16) 0 lambda_3[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_3[0][0]
__________________________________________________________________________________________________
add_3 (Add) (None, 256, 16) 0 add_2[0][0]
conv1d_8[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D) (None, 256, 16) 4112 add_3[0][0]
__________________________________________________________________________________________________
activation_4 (Activation) (None, 256, 16) 0 conv1d_9[0][0]
__________________________________________________________________________________________________
lambda_4 (Lambda) (None, 256, 16) 0 activation_4[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_4 (SpatialDro (None, 256, 16) 0 lambda_4[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_4[0][0]
__________________________________________________________________________________________________
add_4 (Add) (None, 256, 16) 0 add_3[0][0]
conv1d_10[0][0]
__________________________________________________________________________________________________
conv1d_11 (Conv1D) (None, 256, 16) 4112 add_4[0][0]
__________________________________________________________________________________________________
activation_5 (Activation) (None, 256, 16) 0 conv1d_11[0][0]
__________________________________________________________________________________________________
lambda_5 (Lambda) (None, 256, 16) 0 activation_5[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_5 (SpatialDro (None, 256, 16) 0 lambda_5[0][0]
__________________________________________________________________________________________________
conv1d_12 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_5[0][0]
__________________________________________________________________________________________________
add_5 (Add) (None, 256, 16) 0 add_4[0][0]
conv1d_12[0][0]
__________________________________________________________________________________________________
conv1d_13 (Conv1D) (None, 256, 16) 4112 add_5[0][0]
__________________________________________________________________________________________________
activation_6 (Activation) (None, 256, 16) 0 conv1d_13[0][0]
__________________________________________________________________________________________________
lambda_6 (Lambda) (None, 256, 16) 0 activation_6[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_6 (SpatialDro (None, 256, 16) 0 lambda_6[0][0]
__________________________________________________________________________________________________
conv1d_14 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_6[0][0]
__________________________________________________________________________________________________
add_6 (Add) (None, 256, 16) 0 add_5[0][0]
conv1d_14[0][0]
__________________________________________________________________________________________________
conv1d_15 (Conv1D) (None, 256, 16) 4112 add_6[0][0]
__________________________________________________________________________________________________
activation_7 (Activation) (None, 256, 16) 0 conv1d_15[0][0]
__________________________________________________________________________________________________
lambda_7 (Lambda) (None, 256, 16) 0 activation_7[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_7 (SpatialDro (None, 256, 16) 0 lambda_7[0][0]
__________________________________________________________________________________________________
conv1d_16 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_7[0][0]
__________________________________________________________________________________________________
add_7 (Add) (None, 256, 16) 0 add_6[0][0]
conv1d_16[0][0]
__________________________________________________________________________________________________
conv1d_17 (Conv1D) (None, 256, 16) 4112 add_7[0][0]
__________________________________________________________________________________________________
activation_8 (Activation) (None, 256, 16) 0 conv1d_17[0][0]
__________________________________________________________________________________________________
lambda_8 (Lambda) (None, 256, 16) 0 activation_8[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_8 (SpatialDro (None, 256, 16) 0 lambda_8[0][0]
__________________________________________________________________________________________________
conv1d_18 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_8[0][0]
__________________________________________________________________________________________________
add_8 (Add) (None, 256, 16) 0 add_7[0][0]
conv1d_18[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D) (None, 256, 16) 4112 add_8[0][0]
__________________________________________________________________________________________________
activation_9 (Activation) (None, 256, 16) 0 conv1d_19[0][0]
__________________________________________________________________________________________________
lambda_9 (Lambda) (None, 256, 16) 0 activation_9[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_9 (SpatialDro (None, 256, 16) 0 lambda_9[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_9[0][0]
__________________________________________________________________________________________________
add_9 (Add) (None, 256, 16) 0 add_8[0][0]
conv1d_20[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D) (None, 256, 16) 4112 add_9[0][0]
__________________________________________________________________________________________________
activation_10 (Activation) (None, 256, 16) 0 conv1d_21[0][0]
__________________________________________________________________________________________________
lambda_10 (Lambda) (None, 256, 16) 0 activation_10[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_10 (SpatialDr (None, 256, 16) 0 lambda_10[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_10[0][0]
__________________________________________________________________________________________________
add_10 (Add) (None, 256, 16) 0 add_9[0][0]
conv1d_22[0][0]
__________________________________________________________________________________________________
conv1d_23 (Conv1D) (None, 256, 16) 4112 add_10[0][0]
__________________________________________________________________________________________________
activation_11 (Activation) (None, 256, 16) 0 conv1d_23[0][0]
__________________________________________________________________________________________________
lambda_11 (Lambda) (None, 256, 16) 0 activation_11[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_11 (SpatialDr (None, 256, 16) 0 lambda_11[0][0]
__________________________________________________________________________________________________
conv1d_24 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_11[0][0]
__________________________________________________________________________________________________
add_11 (Add) (None, 256, 16) 0 add_10[0][0]
conv1d_24[0][0]
__________________________________________________________________________________________________
conv1d_25 (Conv1D) (None, 256, 16) 4112 add_11[0][0]
__________________________________________________________________________________________________
activation_12 (Activation) (None, 256, 16) 0 conv1d_25[0][0]
__________________________________________________________________________________________________
lambda_12 (Lambda) (None, 256, 16) 0 activation_12[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_12 (SpatialDr (None, 256, 16) 0 lambda_12[0][0]
__________________________________________________________________________________________________
conv1d_26 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_12[0][0]
__________________________________________________________________________________________________
add_12 (Add) (None, 256, 16) 0 add_11[0][0]
conv1d_26[0][0]
__________________________________________________________________________________________________
conv1d_27 (Conv1D) (None, 256, 16) 4112 add_12[0][0]
__________________________________________________________________________________________________
activation_13 (Activation) (None, 256, 16) 0 conv1d_27[0][0]
__________________________________________________________________________________________________
lambda_13 (Lambda) (None, 256, 16) 0 activation_13[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_13 (SpatialDr (None, 256, 16) 0 lambda_13[0][0]
__________________________________________________________________________________________________
conv1d_28 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_13[0][0]
__________________________________________________________________________________________________
add_13 (Add) (None, 256, 16) 0 add_12[0][0]
conv1d_28[0][0]
__________________________________________________________________________________________________
conv1d_29 (Conv1D) (None, 256, 16) 4112 add_13[0][0]
__________________________________________________________________________________________________
activation_14 (Activation) (None, 256, 16) 0 conv1d_29[0][0]
__________________________________________________________________________________________________
lambda_14 (Lambda) (None, 256, 16) 0 activation_14[0][0]
__________________________________________________________________________________________________
spatial_dropout1d_14 (SpatialDr (None, 256, 16) 0 lambda_14[0][0]
__________________________________________________________________________________________________
conv1d_30 (Conv1D) (None, 256, 16) 272 spatial_dropout1d_14[0][0]
__________________________________________________________________________________________________
add_15 (Add) (None, 256, 16) 0 conv1d_2[0][0]
conv1d_4[0][0]
conv1d_6[0][0]
conv1d_8[0][0]
conv1d_10[0][0]
conv1d_12[0][0]
conv1d_14[0][0]
conv1d_16[0][0]
conv1d_18[0][0]
conv1d_20[0][0]
conv1d_22[0][0]
conv1d_24[0][0]
conv1d_26[0][0]
conv1d_28[0][0]
conv1d_30[0][0]
__________________________________________________________________________________________________
activation_15 (Activation) (None, 256, 16) 0 add_15[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 256, 3) 51 activation_15[0][0]
__________________________________________________________________________________________________
activation_16 (Activation) (None, 256, 3) 0 dense[0][0]
==================================================================================================
Total params: 65,843
Trainable params: 65,843
Non-trainable params: 0
__________________________________________________________________________________________________
Epoch 1/4
1000/1000 [==============================] - ETA: 0s - batch: 499.5000 - size: 32.0000 - loss: 0.1143
/Users/janc/miniconda3/lib/python3.8/site-packages/keras/engine/training.py:2470: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
warnings.warn('`Model.state_updates` will be removed in a future version. '
Epoch 00001: val_loss improved from inf to 0.11043, saving model to res/20210924_220702_model.h5
/Users/janc/miniconda3/lib/python3.8/site-packages/keras/utils/generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
warnings.warn('Custom mask layers require a config and must override '
1000/1000 [==============================] - 241s 236ms/step - batch: 499.5000 - size: 32.0000 - loss: 0.1143 - val_loss: 0.1104
Epoch 2/4
1000/1000 [==============================] - ETA: 0s - batch: 499.5000 - size: 32.0000 - loss: 0.0841
Epoch 00002: val_loss improved from 0.11043 to 0.10770, saving model to res/20210924_220702_model.h5
1000/1000 [==============================] - 226s 226ms/step - batch: 499.5000 - size: 32.0000 - loss: 0.0841 - val_loss: 0.1077
Epoch 3/4
1000/1000 [==============================] - ETA: 0s - batch: 499.5000 - size: 32.0000 - loss: 0.0867
Epoch 00003: val_loss did not improve from 0.10770
1000/1000 [==============================] - 215s 215ms/step - batch: 499.5000 - size: 32.0000 - loss: 0.0867 - val_loss: 0.1103
Epoch 4/4
1000/1000 [==============================] - ETA: 0s - batch: 499.5000 - size: 32.0000 - loss: 0.0823
Epoch 00004: val_loss improved from 0.10770 to 0.09988, saving model to res/20210924_220702_model.h5
1000/1000 [==============================] - 218s 218ms/step - batch: 499.5000 - size: 32.0000 - loss: 0.0823 - val_loss: 0.0999
INFO:root:re-loading last best model
INFO:root:predicting
INFO:root:evaluating
INFO:root:[[3545939 7799 38820]
[ 10658 33510 140]
[ 99569 58 241747]]
INFO:root:{'noise': {'precision': 0.9698517518077681, 'recall': 0.9870234523701497, 'f1-score': 0.9783622607234045, 'support': 3592558}, 'pulse': {'precision': 0.8100659946334035, 'recall': 0.7562968312720051, 'f1-score': 0.7822585351619492, 'support': 44308}, 'sine': {'precision': 0.8612075936830931, 'recall': 0.7081587935812335, 'f1-score': 0.7772203298284307, 'support': 341374}, 'accuracy': 0.960524251930502, 'macro avg': {'precision': 0.8803751133747548, 'recall': 0.8171596924077962, 'f1-score': 0.8459470419045948, 'support': 3978240}, 'weighted avg': {'precision': 0.9587493351198522, 'recall': 0.960524251930502, 'f1-score': 0.958918087071358, 'support': 3978240}}
INFO:root:saving to res/20210924_220702_results.h5.
/Users/janc/miniconda3/lib/python3.8/site-packages/tables/attributeset.py:464: NaturalNameWarning: object name is not a valid Python identifier: 'f1-score'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
check_attribute_name(name)
/Users/janc/miniconda3/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'macro avg'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
check_attribute_name(name)
/Users/janc/miniconda3/lib/python3.8/site-packages/tables/path.py:155: NaturalNameWarning: object name is not a valid Python identifier: 'weighted avg'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
check_attribute_name(name)
INFO:root:DONE.
Training using command-line scripts#
The training function das.train.train and all its arguments are also accessible from the command line via das train for use on the terminal. See here for a description of all command-line arguments. The command-line interface is generated with defopt.
For instance, training command above can be invoked from the command line:
das train --data-dir dat/dmel_single_raw.npy --save-dir res --model-name tcn --kernel-size 16 --nb-filters 16 --nb-hist 512 --nb-epoch 20 -i
Shell scripts are particularly useful if you want to fit the network with with different configurations to optimize structural parameters. For instance, this script will fit networks with different numbers of TCN blocks (nb_conv) and filters (nb_filters):
#!/bin/bash
conda activate das
YSUFFIX="pulse"
MODELNAME='tcn'
DATADIR='../dat/dmel_single.npy'
SAVEDIR="res"
NB_HIST=2048
KERNEL_SIZE=32
NB_FILTERS=32
NB_CONV=3
for NB_CONV in 2 3 4
do
for NB_FILTERS in 16 32 64
do
das train -i --nb-filters $NB_FILTERS --kernel-size $KERNEL_SIZE --nb-conv $NB_CONV --nb-hist $NB_HIST --save-dir $SAVEDIR --y-suffix $YSUFFIX --data-dir $DATADIR --model-name $MODELNAME
done
done
A description of all command line arguments can be obtained by typing das train --help in a terminal:
!das train --help
usage: das train [-h] -d DATA_DIR [-y Y_SUFFIX] [--save-dir SAVE_DIR]
[--save-prefix SAVE_PREFIX] [-m MODEL_NAME]
[--nb-filters NB_FILTERS] [-k KERNEL_SIZE]
[--nb-conv NB_CONV] [-u [USE_SEPARABLE [USE_SEPARABLE ...]]]
[--nb-hist NB_HIST]
[-i | --ignore-boundaries | --no-ignore-boundaries]
[--batch-norm | --no-batch-norm] [--nb-pre-conv NB_PRE_CONV]
[--pre-nb-dft PRE_NB_DFT] [--pre-kernel-size PRE_KERNEL_SIZE]
[--pre-nb-filters PRE_NB_FILTERS] [--pre-nb-conv PRE_NB_CONV]
[--nb-lstm-units NB_LSTM_UNITS] [--verbose VERBOSE]
[--batch-size BATCH_SIZE] [--nb-epoch NB_EPOCH]
[--learning-rate LEARNING_RATE]
[--reduce-lr | --no-reduce-lr]
[--reduce-lr-patience REDUCE_LR_PATIENCE] [-f FRACTION_DATA]
[--seed SEED]
[--batch-level-subsampling | --no-batch-level-subsampling]
[-t | --tensorboard | --no-tensorboard]
[--neptune-api-token NEPTUNE_API_TOKEN]
[--neptune-project NEPTUNE_PROJECT]
[--log-messages | --no-log-messages] [--nb-stacks NB_STACKS]
[-w | --with-y-hist | --no-with-y-hist] [-x X_SUFFIX]
[--balance | --no-balance]
[--version-data | --no-version-data]
Train a DeepSS network.
optional arguments:
-h, --help show this help message and exit
-d DATA_DIR, --data-dir DATA_DIR
Path to the directory or file with the dataset for training.
Accepts npy-dirs (recommended), h5 files or zarr files.
See documentation for how the dataset should be organized.
-y Y_SUFFIX, --y-suffix Y_SUFFIX
Select training target by suffix.
Song-type specific targets can be created with a training dataset,
Defaults to '' (will use the standard target 'y')
--save-dir SAVE_DIR Directory to save training outputs.
The path of output files will constructed from the SAVE_DIR, an optional prefix, and the time stamp of the start of training.
Defaults to current directory ('./').
--save-prefix SAVE_PREFIX
Prepend to timestamp.
Name of files created will be SAVE_DIR/SAVE_PREFIX + "_" + TIMESTAMP
or SAVE_DIR/ TIMESTAMP if SAVE_PREFIX is empty.
Defaults to '' (empty).
-m MODEL_NAME, --model-name MODEL_NAME
Network architecture to use.
Use "tcn" (TCN) or "tcn_stft" (TCN with STFT frontend).
See das.models for a description of all models.
Defaults to 'tcn'.
--nb-filters NB_FILTERS
Number of filters per layer.
Defaults to 16.
-k KERNEL_SIZE, --kernel-size KERNEL_SIZE
Duration of the filters (=kernels) in samples.
Defaults to 16.
--nb-conv NB_CONV Number of TCN blocks in the network.
Defaults to 3.
-u [USE_SEPARABLE [USE_SEPARABLE ...]], --use-separable [USE_SEPARABLE [USE_SEPARABLE ...]]
Specify which TCN blocks should use separable convolutions.
Provide as a space-separated sequence of "False" or "True.
For instance: "True False False" will set the first block in a
three-block (as given by nb_conv) network to use separable convolutions.
Defaults to False (no block uses separable convolution).
--nb-hist NB_HIST Number of samples processed at once by the network (a.k.a chunk size).
Defaults to 1024.
-i, --ignore-boundaries, --no-ignore-boundaries
Minimize edge effects by discarding predictions at the edges of chunks.
Defaults to True.
--batch-norm, --no-batch-norm
Batch normalize.
Defaults to True.
--nb-pre-conv NB_PRE_CONV
Downsampling rate. Adds downsampling frontend if not 0.
TCN_TCN: adds a frontend of N conv blocks (conv-relu-batchnorm-maxpool2) to the TCN.
TCN_STFT: adds a trainable STFT frontend.
Defaults to 0 (no frontend).
--pre-nb-dft PRE_NB_DFT
Number of filters (roughly corresponding to filters) in the STFT frontend.
Defaults to 64.
--pre-kernel-size PRE_KERNEL_SIZE
Duration of filters (=kernels) in samples in the pre-processing TCN.
Defaults to 3.
--pre-nb-filters PRE_NB_FILTERS
Number of filters per layer in the pre-processing TCN.
Defaults to 16.
--pre-nb-conv PRE_NB_CONV
--nb-lstm-units NB_LSTM_UNITS
If >0, adds LSTM with given number of units to the output of the stack of TCN blocks.
Defaults to 0 (no LSTM layer).
--verbose VERBOSE Verbosity of training output (0 - no output(?), 1 - progress bar, 2 - one line per epoch).
Defaults to 2.
--batch-size BATCH_SIZE
Batch size
Defaults to 32.
--nb-epoch NB_EPOCH Maximal number of training epochs.
Training will stop early if validation loss did not decrease in the last 20 epochs.
Defaults to 400.
--learning-rate LEARNING_RATE
Learning rate of the model. Defaults should work in most cases.
Values typically range between 0.1 and 0.00001.
If None, uses per model defaults: "tcn" 0.0001, "tcn_stft" 0.0005).
Defaults to None.
--reduce-lr, --no-reduce-lr
Reduce learning rate on plateau.
Defaults to False.
--reduce-lr-patience REDUCE_LR_PATIENCE
Number of epochs w/o a reduction in validation loss after which to trigger a reduction in learning rate.
Defaults to 5.
-f FRACTION_DATA, --fraction-data FRACTION_DATA
Fraction of training and validation to use for training.
Defaults to 1.0.
--seed SEED Random seed to reproducible select fractions of the data.
Defaults to None (no seed).
--batch-level-subsampling, --no-batch-level-subsampling
Select fraction of data for training from random subset of shuffled batches.
If False, select a continuous chunk of the recording.
Defaults to False.
-t, --tensorboard, --no-tensorboard
Write tensorboard logs to save_dir. Defaults to False.
--neptune-api-token NEPTUNE_API_TOKEN
API token for logging to neptune.ai. Defaults to None (no logging).
--neptune-project NEPTUNE_PROJECT
Project to log to for neptune.ai. Defaults to None (no logging).
--log-messages, --no-log-messages
Sets logging level to INFO.
Defaults to False (will follow existing settings).
--nb-stacks NB_STACKS
Unused if model name is "tcn" or "tcn_stft". Defaults to 2.
-w, --with-y-hist, --no-with-y-hist
Unused if model name is "tcn" or "tcn_stft". Defaults to True.
-x X_SUFFIX, --x-suffix X_SUFFIX
Select specific training data based on suffix (e.g. x_suffix).
Defaults to '' (will use the standard data 'x')
--balance, --no-balance
Balance data. Weights class-wise errors by the inverse of the class frequencies.
Defaults to False.
--version-data, --no-version-data
Save MD5 hash of the data_dir to log and params.yaml.
Defaults to True (set to False for large datasets since it can be slow).