Train

The network can be trained using three interfaces:

  • python, via das.train.train

  • the command-line interface das train.

  • the GUI - see the GUI tutorial

Training will:

  • load train/val/test data form a dataset

  • initialize the network

  • save all parameters for reproducibility

  • train the network and save the best network to disk

  • run inference and evaluate the network using the test data.

The names of files created during training start with an optional prefix and the time stamp of the start time of training, as in my-awesome-prefix_20192310_091032. Typically, three files are created:

  • *_params.yaml - training parameters etc.

  • *_model.h5 - model architecture and weights

  • *_results.h5 - predictions and evaluation results for the test set (only created if the training dataset contains a test set)

Training using python

Training is done using the train function in the das.train module:

import das.train
help(das.train.train)
Help on function train in module das.train:

train(*, data_dir: str, y_suffix: str = '', save_dir: str = './', save_prefix: Union[str, NoneType] = None, model_name: str = 'tcn', nb_filters: int = 16, kernel_size: int = 16, nb_conv: int = 3, use_separable: List[bool] = False, nb_hist: int = 1024, ignore_boundaries: bool = True, batch_norm: bool = True, nb_pre_conv: int = 0, pre_kernel_size: int = 3, pre_nb_filters: int = 16, pre_nb_conv: int = 2, verbose: int = 2, batch_size: int = 32, nb_epoch: int = 400, learning_rate: Union[float, NoneType] = None, reduce_lr: bool = False, reduce_lr_patience: int = 5, fraction_data: Union[float, NoneType] = None, seed: Union[int, NoneType] = None, batch_level_subsampling: bool = False, tensorboard: bool = False, log_messages: bool = False, nb_stacks: int = 2, with_y_hist: bool = True, x_suffix: str = '', _qt_progress: bool = False)
    Train a DeepSS network.
    
    Args:
        data_dir (str): Path to the directory or file with the dataset for training.
                        Accepts npy-dirs (recommended), h5 files or zarr files.
                        See documentation for how the dataset should be organized.
        y_suffix (str): Select training target by suffix.
                        Song-type specific targets can be created with a training dataset,
                        Defaults to '' (will use the standard target 'y')
        save_dir (str): Directory to save training outputs.
                        The path of output files will constructed from the SAVE_DIR, an optional prefix, and the time stamp of the start of training.
                        Defaults to current directory ('./').
        save_prefix (Optional[str]): Prepend to timestamp.
                           Name of files created will be SAVE_DIR/SAVE_PREFIX + "_" + TIMESTAMP
                           or SAVE_DIR/ TIMESTAMP if SAVE_PREFIX is empty.
                           Defaults to '' (empty).
        model_name (str): Network architecture to use.
                          Use "tcn" (TCN) or "tcn_stft" (TCN with STFT frontend).
                          See das.models for a description of all models.
                          Defaults to 'tcn'.
        nb_filters (int): Number of filters per layer.
                          Defaults to 16.
        kernel_size (int): Duration of the filters (=kernels) in samples.
                           Defaults to 16.
        nb_conv (int): Number of TCN blocks in the network.
                       Defaults to 3.
        use_separable (List[bool]): Specify which TCN blocks should use separable convolutions.
                                    Provide as a space-separated sequence of "False" or "True.
                                    For instance: "True False False" will set the first block in a
                                    three-block (as given by nb_conv) network to use separable convolutions.
                                    Defaults to False (no block uses separable convolution).
        nb_hist (int): Number of samples processed at once by the network (a.k.a chunk size).
                       Defaults to 1024.
        ignore_boundaries (bool): Minimize edge effects by discarding predictions at the edges of chunks.
                                  Defaults to True.
        batch_norm (bool): Batch normalize.
                           Defaults to True.
        nb_pre_conv (int): Adds downsampling frontend.
                           TCN: adds a frontend of N conv blocks (conv-relu-batchnorm-maxpool2) to the TCN - useful for reducing the sampling rate for USV.
                           TCN_STFT: stft
                           Defaults to 0 (no frontend).
        pre_nb_filters (int): [description]. Defaults to 16.
        pre_kernel_size (int): [description]. Defaults to 3.
        pre_nb_conv (int): [description]. Defaults to 3.
        verbose (int): Verbosity of training output (0 - no output(?), 1 - progress bar, 2 - one line per epoch).
                       Defaults to 2.
        batch_size (int): Batch size
                          Defaults to 32.
        nb_epoch (int): Maximal number of training epochs.
                        Training will stop early if validation loss did not decrease in the last 20 epochs.
                        Defaults to 400.
        learning_rate (Optional[float]): Learning rate of the model. Defaults should work in most cases.
                               Values typically range between 0.1 and 0.00001.
                               If None, uses per model defaults: "tcn" 0.0001, "tcn_stft" 0.0005).
                               Defaults to None.
        reduce_lr (bool): Reduce learning rate on plateau.
                          Defaults to False.
        reduce_lr_patience (int): Number of epochs w/o a reduction in validation loss after which to trigger a reduction in learning rate.
                                  Defaults to 5.
        fraction_data (Optional[float]): Fraction of training and validation to use for training.
                               Defaults to 1.0.
        seed (Optional[int]): Random seed to reproducible select fractions of the data.
                    Defaults to None (no seed).
        batch_level_subsampling (bool): Select fraction of data for training from random subset of shuffled batches.
                                        If False, select a continuous chunk of the recording.
                                        Defaults to False.
        tensorboard (bool): Write tensorboard logs to save_dir.
                            Defaults to False.
        log_messages (bool): Sets logging level to INFO.
                             Defaults to False (will follow existing settings).
        nb_stacks (int): Unused if model name is "tcn" or "tcn_stft". Defaults to 2.
        with_y_hist (bool): Unused if model name is "tcn" or "tcn_stft". Defaults to True.
        x_suffix (str): Select specific training data based on suffix (e.g. x_suffix).
                        Defaults to '' (will use the standard data 'x')

The output of the train function is fairly verbose, to help with troubleshooting:

  • run time parameters

  • information on the size of the training and validation data

  • network architecture

  • training progress (training and validation loss)

  • after training, a classification report for the test data (if test data exist in the dataset)

To demonstrate the outputs of train, the following trains a small network on a small dataset to annotate pulse and sine song from Drosophila melanogater. Expected performance (f1-score) is about 75%.

das.train.train(model_name='tcn',  # see `das.models` for valid model_names
                data_dir='tutorial_dataset.npy', 
                save_dir='res',
                nb_hist=256,
                kernel_size=16,
                nb_filters=16,
                ignore_boundaries=True,
                verbose=1,
                nb_epoch=4,
                log_messages=True)
INFO:root:loading data
INFO:root:Parameters:
INFO:root:{'data_dir': 'tutorial_dataset.npy', 'y_suffix': '', 'save_dir': 'res', 'save_prefix': '', 'model_name': 'tcn', 'nb_filters': 16, 'kernel_size': 16, 'nb_conv': 3, 'use_separable': False, 'nb_hist': 256, 'ignore_boundaries': True, 'batch_norm': True, 'nb_pre_conv': 0, 'pre_kernel_size': 3, 'pre_nb_filters': 16, 'pre_nb_conv': 2, 'verbose': 1, 'batch_size': 32, 'nb_epoch': 4, 'reduce_lr': False, 'reduce_lr_patience': 5, 'fraction_data': None, 'seed': None, 'batch_level_subsampling': False, 'tensorboard': False, 'log_messages': True, 'nb_stacks': 2, 'with_y_hist': True, 'x_suffix': '', 'sample_weight_mode': 'temporal', 'data_padding': 48, 'return_sequences': True, 'stride': 160, 'y_offset': 0, 'output_stride': 1, 'class_names': ['noise', 'pulse', 'sine'], 'class_names_pulse': ['noise', 'pulse'], 'class_names_sine': ['noise', 'sine'], 'class_types': ['segment', 'event', 'segment'], 'class_types_pulse': ['segment', 'event'], 'class_types_sine': ['segment', 'segment'], 'filename_endsample_test': [], 'filename_endsample_train': [], 'filename_endsample_val': [], 'filename_startsample_test': [], 'filename_startsample_train': [], 'filename_startsample_val': [], 'filename_train': [], 'filename_val': [], 'samplerate_x_Hz': 10000, 'samplerate_y_Hz': 10000, 'filename_test': [], 'nb_freq': 1, 'nb_channels': 1, 'nb_classes': 3, 'first_sample_train': 0, 'last_sample_train': None, 'first_sample_val': 0, 'last_sample_val': None}
INFO:root:preparing data
INFO:root:Training data:
INFO:root:AudioSequence with 3992 batches each with 32 items.
   Total of 20440005 samples with
   each x=(1,) and
   each y=(3,)
INFO:root:Validation data:
INFO:root:AudioSequence with 812 batches each with 32 items.
   Total of 4160001 samples with
   each x=(1,) and
   each y=(3,)
INFO:root:building network
WARNING:tensorflow:From /Users/clemens10/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /Users/clemens10/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:root:None
INFO:root:Will save to res/20210622_101812.
Model: "TCN"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 256, 1)]     0                                            
__________________________________________________________________________________________________
conv1d (Conv1D)                 (None, 256, 16)      32          input_1[0][0]                    
__________________________________________________________________________________________________
conv1d_1 (Conv1D)               (None, 256, 16)      4112        conv1d[0][0]                     
__________________________________________________________________________________________________
activation (Activation)         (None, 256, 16)      0           conv1d_1[0][0]                   
__________________________________________________________________________________________________
lambda (Lambda)                 (None, 256, 16)      0           activation[0][0]                 
__________________________________________________________________________________________________
spatial_dropout1d (SpatialDropo (None, 256, 16)      0           lambda[0][0]                     
__________________________________________________________________________________________________
conv1d_2 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d[0][0]          
__________________________________________________________________________________________________
add (Add)                       (None, 256, 16)      0           conv1d[0][0]                     
                                                                 conv1d_2[0][0]                   
__________________________________________________________________________________________________
conv1d_3 (Conv1D)               (None, 256, 16)      4112        add[0][0]                        
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 256, 16)      0           conv1d_3[0][0]                   
__________________________________________________________________________________________________
lambda_1 (Lambda)               (None, 256, 16)      0           activation_1[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_1 (SpatialDro (None, 256, 16)      0           lambda_1[0][0]                   
__________________________________________________________________________________________________
conv1d_4 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_1[0][0]        
__________________________________________________________________________________________________
add_1 (Add)                     (None, 256, 16)      0           add[0][0]                        
                                                                 conv1d_4[0][0]                   
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 256, 16)      4112        add_1[0][0]                      
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 256, 16)      0           conv1d_5[0][0]                   
__________________________________________________________________________________________________
lambda_2 (Lambda)               (None, 256, 16)      0           activation_2[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_2 (SpatialDro (None, 256, 16)      0           lambda_2[0][0]                   
__________________________________________________________________________________________________
conv1d_6 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_2[0][0]        
__________________________________________________________________________________________________
add_2 (Add)                     (None, 256, 16)      0           add_1[0][0]                      
                                                                 conv1d_6[0][0]                   
__________________________________________________________________________________________________
conv1d_7 (Conv1D)               (None, 256, 16)      4112        add_2[0][0]                      
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 256, 16)      0           conv1d_7[0][0]                   
__________________________________________________________________________________________________
lambda_3 (Lambda)               (None, 256, 16)      0           activation_3[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_3 (SpatialDro (None, 256, 16)      0           lambda_3[0][0]                   
__________________________________________________________________________________________________
conv1d_8 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_3[0][0]        
__________________________________________________________________________________________________
add_3 (Add)                     (None, 256, 16)      0           add_2[0][0]                      
                                                                 conv1d_8[0][0]                   
__________________________________________________________________________________________________
conv1d_9 (Conv1D)               (None, 256, 16)      4112        add_3[0][0]                      
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 256, 16)      0           conv1d_9[0][0]                   
__________________________________________________________________________________________________
lambda_4 (Lambda)               (None, 256, 16)      0           activation_4[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_4 (SpatialDro (None, 256, 16)      0           lambda_4[0][0]                   
__________________________________________________________________________________________________
conv1d_10 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_4[0][0]        
__________________________________________________________________________________________________
add_4 (Add)                     (None, 256, 16)      0           add_3[0][0]                      
                                                                 conv1d_10[0][0]                  
__________________________________________________________________________________________________
conv1d_11 (Conv1D)              (None, 256, 16)      4112        add_4[0][0]                      
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 256, 16)      0           conv1d_11[0][0]                  
__________________________________________________________________________________________________
lambda_5 (Lambda)               (None, 256, 16)      0           activation_5[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_5 (SpatialDro (None, 256, 16)      0           lambda_5[0][0]                   
__________________________________________________________________________________________________
conv1d_12 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_5[0][0]        
__________________________________________________________________________________________________
add_5 (Add)                     (None, 256, 16)      0           add_4[0][0]                      
                                                                 conv1d_12[0][0]                  
__________________________________________________________________________________________________
conv1d_13 (Conv1D)              (None, 256, 16)      4112        add_5[0][0]                      
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 256, 16)      0           conv1d_13[0][0]                  
__________________________________________________________________________________________________
lambda_6 (Lambda)               (None, 256, 16)      0           activation_6[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_6 (SpatialDro (None, 256, 16)      0           lambda_6[0][0]                   
__________________________________________________________________________________________________
conv1d_14 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_6[0][0]        
__________________________________________________________________________________________________
add_6 (Add)                     (None, 256, 16)      0           add_5[0][0]                      
                                                                 conv1d_14[0][0]                  
__________________________________________________________________________________________________
conv1d_15 (Conv1D)              (None, 256, 16)      4112        add_6[0][0]                      
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 256, 16)      0           conv1d_15[0][0]                  
__________________________________________________________________________________________________
lambda_7 (Lambda)               (None, 256, 16)      0           activation_7[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_7 (SpatialDro (None, 256, 16)      0           lambda_7[0][0]                   
__________________________________________________________________________________________________
conv1d_16 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_7[0][0]        
__________________________________________________________________________________________________
add_7 (Add)                     (None, 256, 16)      0           add_6[0][0]                      
                                                                 conv1d_16[0][0]                  
__________________________________________________________________________________________________
conv1d_17 (Conv1D)              (None, 256, 16)      4112        add_7[0][0]                      
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 256, 16)      0           conv1d_17[0][0]                  
__________________________________________________________________________________________________
lambda_8 (Lambda)               (None, 256, 16)      0           activation_8[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_8 (SpatialDro (None, 256, 16)      0           lambda_8[0][0]                   
__________________________________________________________________________________________________
conv1d_18 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_8[0][0]        
__________________________________________________________________________________________________
add_8 (Add)                     (None, 256, 16)      0           add_7[0][0]                      
                                                                 conv1d_18[0][0]                  
__________________________________________________________________________________________________
conv1d_19 (Conv1D)              (None, 256, 16)      4112        add_8[0][0]                      
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 256, 16)      0           conv1d_19[0][0]                  
__________________________________________________________________________________________________
lambda_9 (Lambda)               (None, 256, 16)      0           activation_9[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_9 (SpatialDro (None, 256, 16)      0           lambda_9[0][0]                   
__________________________________________________________________________________________________
conv1d_20 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_9[0][0]        
__________________________________________________________________________________________________
add_9 (Add)                     (None, 256, 16)      0           add_8[0][0]                      
                                                                 conv1d_20[0][0]                  
__________________________________________________________________________________________________
conv1d_21 (Conv1D)              (None, 256, 16)      4112        add_9[0][0]                      
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 256, 16)      0           conv1d_21[0][0]                  
__________________________________________________________________________________________________
lambda_10 (Lambda)              (None, 256, 16)      0           activation_10[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_10 (SpatialDr (None, 256, 16)      0           lambda_10[0][0]                  
__________________________________________________________________________________________________
conv1d_22 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_10[0][0]       
__________________________________________________________________________________________________
add_10 (Add)                    (None, 256, 16)      0           add_9[0][0]                      
                                                                 conv1d_22[0][0]                  
__________________________________________________________________________________________________
conv1d_23 (Conv1D)              (None, 256, 16)      4112        add_10[0][0]                     
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 256, 16)      0           conv1d_23[0][0]                  
__________________________________________________________________________________________________
lambda_11 (Lambda)              (None, 256, 16)      0           activation_11[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_11 (SpatialDr (None, 256, 16)      0           lambda_11[0][0]                  
__________________________________________________________________________________________________
conv1d_24 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_11[0][0]       
__________________________________________________________________________________________________
add_11 (Add)                    (None, 256, 16)      0           add_10[0][0]                     
                                                                 conv1d_24[0][0]                  
__________________________________________________________________________________________________
conv1d_25 (Conv1D)              (None, 256, 16)      4112        add_11[0][0]                     
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 256, 16)      0           conv1d_25[0][0]                  
__________________________________________________________________________________________________
lambda_12 (Lambda)              (None, 256, 16)      0           activation_12[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_12 (SpatialDr (None, 256, 16)      0           lambda_12[0][0]                  
__________________________________________________________________________________________________
conv1d_26 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_12[0][0]       
__________________________________________________________________________________________________
add_12 (Add)                    (None, 256, 16)      0           add_11[0][0]                     
                                                                 conv1d_26[0][0]                  
__________________________________________________________________________________________________
conv1d_27 (Conv1D)              (None, 256, 16)      4112        add_12[0][0]                     
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 256, 16)      0           conv1d_27[0][0]                  
__________________________________________________________________________________________________
lambda_13 (Lambda)              (None, 256, 16)      0           activation_13[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_13 (SpatialDr (None, 256, 16)      0           lambda_13[0][0]                  
__________________________________________________________________________________________________
conv1d_28 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_13[0][0]       
__________________________________________________________________________________________________
add_13 (Add)                    (None, 256, 16)      0           add_12[0][0]                     
                                                                 conv1d_28[0][0]                  
__________________________________________________________________________________________________
conv1d_29 (Conv1D)              (None, 256, 16)      4112        add_13[0][0]                     
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 256, 16)      0           conv1d_29[0][0]                  
__________________________________________________________________________________________________
lambda_14 (Lambda)              (None, 256, 16)      0           activation_14[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_14 (SpatialDr (None, 256, 16)      0           lambda_14[0][0]                  
__________________________________________________________________________________________________
conv1d_30 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_14[0][0]       
__________________________________________________________________________________________________
add_15 (Add)                    (None, 256, 16)      0           conv1d_2[0][0]                   
                                                                 conv1d_4[0][0]                   
                                                                 conv1d_6[0][0]                   
                                                                 conv1d_8[0][0]                   
                                                                 conv1d_10[0][0]                  
                                                                 conv1d_12[0][0]                  
                                                                 conv1d_14[0][0]                  
                                                                 conv1d_16[0][0]                  
                                                                 conv1d_18[0][0]                  
                                                                 conv1d_20[0][0]                  
                                                                 conv1d_22[0][0]                  
                                                                 conv1d_24[0][0]                  
                                                                 conv1d_26[0][0]                  
                                                                 conv1d_28[0][0]                  
                                                                 conv1d_30[0][0]                  
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 256, 16)      0           add_15[0][0]                     
__________________________________________________________________________________________________
dense (Dense)                   (None, 256, 3)       51          activation_15[0][0]              
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 256, 3)       0           dense[0][0]                      
==================================================================================================
Total params: 65,843
Trainable params: 65,843
Non-trainable params: 0
__________________________________________________________________________________________________
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-90b7bda8dbbb> in <module>
      8                 verbose=1,
      9                 nb_epoch=4,
---> 10                 log_messages=True)

/Volumes/shinkansen/Dropbox/code.py/das/src/das/train.py in train(data_dir, y_suffix, save_dir, save_prefix, model_name, nb_filters, kernel_size, nb_conv, use_separable, nb_hist, ignore_boundaries, batch_norm, nb_pre_conv, pre_kernel_size, pre_nb_filters, pre_nb_conv, verbose, batch_size, nb_epoch, learning_rate, reduce_lr, reduce_lr_patience, fraction_data, seed, batch_level_subsampling, tensorboard, log_messages, nb_stacks, with_y_hist, x_suffix, _qt_progress)
    223 
    224     if _qt_progress is not None:
--> 225         callbacks.append(utils.QtProgressCallback(nb_epoch, _qt_progress))
    226 
    227     if tensorboard:

/Volumes/shinkansen/Dropbox/code.py/das/src/das/utils.py in __init__(self, nb_epochs, comms)
    187         super().__init__()
    188         self.nb_epochs = nb_epochs
--> 189         self.queue = comms[0]
    190         self.stop_event = comms[1]
    191 

TypeError: 'bool' object is not subscriptable
INFO:root:loading data
INFO:root:Parameters:
INFO:root:{'data_dir': 'tutorial_dataset.npy', 'y_suffix': '', 'save_dir': 'res', 'save_prefix': '', 'model_name': 'tcn', 'nb_filters': 16, 'kernel_size': 16, 'nb_conv': 3, 'use_separable': False, 'nb_hist': 256, 'ignore_boundaries': True, 'batch_norm': True, 'nb_pre_conv': 0, 'pre_kernel_size': 3, 'pre_nb_filters': 16, 'pre_nb_conv': 2, 'verbose': 1, 'batch_size': 32, 'nb_epoch': 4, 'reduce_lr': False, 'reduce_lr_patience': 5, 'fraction_data': None, 'seed': None, 'batch_level_subsampling': False, 'tensorboard': False, 'log_messages': True, 'nb_stacks': 2, 'with_y_hist': True, 'x_suffix': '', 'sample_weight_mode': 'temporal', 'data_padding': 48, 'return_sequences': True, 'stride': 160, 'y_offset': 0, 'output_stride': 1, 'class_names': ['noise', 'pulse', 'sine'], 'class_names_pulse': ['noise', 'pulse'], 'class_names_sine': ['noise', 'sine'], 'class_types': ['segment', 'event', 'segment'], 'class_types_pulse': ['segment', 'event'], 'class_types_sine': ['segment', 'segment'], 'filename_endsample_test': [], 'filename_endsample_train': [], 'filename_endsample_val': [], 'filename_startsample_test': [], 'filename_startsample_train': [], 'filename_startsample_val': [], 'filename_train': [], 'filename_val': [], 'samplerate_x_Hz': 10000, 'samplerate_y_Hz': 10000, 'filename_test': [], 'nb_freq': 1, 'nb_channels': 1, 'nb_classes': 3, 'first_sample_train': 0, 'last_sample_train': None, 'first_sample_val': 0, 'last_sample_val': None}
INFO:root:preparing data
INFO:root:Training data:
INFO:root:AudioSequence with 3992 batches each with 32 items.
   Total of 20440005 samples with
   each x=(1,) and
   each y=(3,)
INFO:root:Validation data:
INFO:root:AudioSequence with 812 batches each with 32 items.
   Total of 4160001 samples with
   each x=(1,) and
   each y=(3,)
INFO:root:building network
WARNING:tensorflow:From /Users/clemens10/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /Users/clemens10/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:root:None
INFO:root:start training
Model: "TCN"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 256, 1)]     0                                            
__________________________________________________________________________________________________
conv1d (Conv1D)                 (None, 256, 16)      32          input_1[0][0]                    
__________________________________________________________________________________________________
conv1d_1 (Conv1D)               (None, 256, 16)      4112        conv1d[0][0]                     
__________________________________________________________________________________________________
activation (Activation)         (None, 256, 16)      0           conv1d_1[0][0]                   
__________________________________________________________________________________________________
lambda (Lambda)                 (None, 256, 16)      0           activation[0][0]                 
__________________________________________________________________________________________________
spatial_dropout1d (SpatialDropo (None, 256, 16)      0           lambda[0][0]                     
__________________________________________________________________________________________________
conv1d_2 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d[0][0]          
__________________________________________________________________________________________________
add (Add)                       (None, 256, 16)      0           conv1d[0][0]                     
                                                                 conv1d_2[0][0]                   
__________________________________________________________________________________________________
conv1d_3 (Conv1D)               (None, 256, 16)      4112        add[0][0]                        
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 256, 16)      0           conv1d_3[0][0]                   
__________________________________________________________________________________________________
lambda_1 (Lambda)               (None, 256, 16)      0           activation_1[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_1 (SpatialDro (None, 256, 16)      0           lambda_1[0][0]                   
__________________________________________________________________________________________________
conv1d_4 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_1[0][0]        
__________________________________________________________________________________________________
add_1 (Add)                     (None, 256, 16)      0           add[0][0]                        
                                                                 conv1d_4[0][0]                   
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 256, 16)      4112        add_1[0][0]                      
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 256, 16)      0           conv1d_5[0][0]                   
__________________________________________________________________________________________________
lambda_2 (Lambda)               (None, 256, 16)      0           activation_2[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_2 (SpatialDro (None, 256, 16)      0           lambda_2[0][0]                   
__________________________________________________________________________________________________
conv1d_6 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_2[0][0]        
__________________________________________________________________________________________________
add_2 (Add)                     (None, 256, 16)      0           add_1[0][0]                      
                                                                 conv1d_6[0][0]                   
__________________________________________________________________________________________________
conv1d_7 (Conv1D)               (None, 256, 16)      4112        add_2[0][0]                      
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 256, 16)      0           conv1d_7[0][0]                   
__________________________________________________________________________________________________
lambda_3 (Lambda)               (None, 256, 16)      0           activation_3[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_3 (SpatialDro (None, 256, 16)      0           lambda_3[0][0]                   
__________________________________________________________________________________________________
conv1d_8 (Conv1D)               (None, 256, 16)      272         spatial_dropout1d_3[0][0]        
__________________________________________________________________________________________________
add_3 (Add)                     (None, 256, 16)      0           add_2[0][0]                      
                                                                 conv1d_8[0][0]                   
__________________________________________________________________________________________________
conv1d_9 (Conv1D)               (None, 256, 16)      4112        add_3[0][0]                      
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 256, 16)      0           conv1d_9[0][0]                   
__________________________________________________________________________________________________
lambda_4 (Lambda)               (None, 256, 16)      0           activation_4[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_4 (SpatialDro (None, 256, 16)      0           lambda_4[0][0]                   
__________________________________________________________________________________________________
conv1d_10 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_4[0][0]        
__________________________________________________________________________________________________
add_4 (Add)                     (None, 256, 16)      0           add_3[0][0]                      
                                                                 conv1d_10[0][0]                  
__________________________________________________________________________________________________
conv1d_11 (Conv1D)              (None, 256, 16)      4112        add_4[0][0]                      
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 256, 16)      0           conv1d_11[0][0]                  
__________________________________________________________________________________________________
lambda_5 (Lambda)               (None, 256, 16)      0           activation_5[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_5 (SpatialDro (None, 256, 16)      0           lambda_5[0][0]                   
__________________________________________________________________________________________________
conv1d_12 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_5[0][0]        
__________________________________________________________________________________________________
add_5 (Add)                     (None, 256, 16)      0           add_4[0][0]                      
                                                                 conv1d_12[0][0]                  
__________________________________________________________________________________________________
conv1d_13 (Conv1D)              (None, 256, 16)      4112        add_5[0][0]                      
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 256, 16)      0           conv1d_13[0][0]                  
__________________________________________________________________________________________________
lambda_6 (Lambda)               (None, 256, 16)      0           activation_6[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_6 (SpatialDro (None, 256, 16)      0           lambda_6[0][0]                   
__________________________________________________________________________________________________
conv1d_14 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_6[0][0]        
__________________________________________________________________________________________________
add_6 (Add)                     (None, 256, 16)      0           add_5[0][0]                      
                                                                 conv1d_14[0][0]                  
__________________________________________________________________________________________________
conv1d_15 (Conv1D)              (None, 256, 16)      4112        add_6[0][0]                      
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 256, 16)      0           conv1d_15[0][0]                  
__________________________________________________________________________________________________
lambda_7 (Lambda)               (None, 256, 16)      0           activation_7[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_7 (SpatialDro (None, 256, 16)      0           lambda_7[0][0]                   
__________________________________________________________________________________________________
conv1d_16 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_7[0][0]        
__________________________________________________________________________________________________
add_7 (Add)                     (None, 256, 16)      0           add_6[0][0]                      
                                                                 conv1d_16[0][0]                  
__________________________________________________________________________________________________
conv1d_17 (Conv1D)              (None, 256, 16)      4112        add_7[0][0]                      
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 256, 16)      0           conv1d_17[0][0]                  
__________________________________________________________________________________________________
lambda_8 (Lambda)               (None, 256, 16)      0           activation_8[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_8 (SpatialDro (None, 256, 16)      0           lambda_8[0][0]                   
__________________________________________________________________________________________________
conv1d_18 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_8[0][0]        
__________________________________________________________________________________________________
add_8 (Add)                     (None, 256, 16)      0           add_7[0][0]                      
                                                                 conv1d_18[0][0]                  
__________________________________________________________________________________________________
conv1d_19 (Conv1D)              (None, 256, 16)      4112        add_8[0][0]                      
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 256, 16)      0           conv1d_19[0][0]                  
__________________________________________________________________________________________________
lambda_9 (Lambda)               (None, 256, 16)      0           activation_9[0][0]               
__________________________________________________________________________________________________
spatial_dropout1d_9 (SpatialDro (None, 256, 16)      0           lambda_9[0][0]                   
__________________________________________________________________________________________________
conv1d_20 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_9[0][0]        
__________________________________________________________________________________________________
add_9 (Add)                     (None, 256, 16)      0           add_8[0][0]                      
                                                                 conv1d_20[0][0]                  
__________________________________________________________________________________________________
conv1d_21 (Conv1D)              (None, 256, 16)      4112        add_9[0][0]                      
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 256, 16)      0           conv1d_21[0][0]                  
__________________________________________________________________________________________________
lambda_10 (Lambda)              (None, 256, 16)      0           activation_10[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_10 (SpatialDr (None, 256, 16)      0           lambda_10[0][0]                  
__________________________________________________________________________________________________
conv1d_22 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_10[0][0]       
__________________________________________________________________________________________________
add_10 (Add)                    (None, 256, 16)      0           add_9[0][0]                      
                                                                 conv1d_22[0][0]                  
__________________________________________________________________________________________________
conv1d_23 (Conv1D)              (None, 256, 16)      4112        add_10[0][0]                     
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 256, 16)      0           conv1d_23[0][0]                  
__________________________________________________________________________________________________
lambda_11 (Lambda)              (None, 256, 16)      0           activation_11[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_11 (SpatialDr (None, 256, 16)      0           lambda_11[0][0]                  
__________________________________________________________________________________________________
conv1d_24 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_11[0][0]       
__________________________________________________________________________________________________
add_11 (Add)                    (None, 256, 16)      0           add_10[0][0]                     
                                                                 conv1d_24[0][0]                  
__________________________________________________________________________________________________
conv1d_25 (Conv1D)              (None, 256, 16)      4112        add_11[0][0]                     
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 256, 16)      0           conv1d_25[0][0]                  
__________________________________________________________________________________________________
lambda_12 (Lambda)              (None, 256, 16)      0           activation_12[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_12 (SpatialDr (None, 256, 16)      0           lambda_12[0][0]                  
__________________________________________________________________________________________________
conv1d_26 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_12[0][0]       
__________________________________________________________________________________________________
add_12 (Add)                    (None, 256, 16)      0           add_11[0][0]                     
                                                                 conv1d_26[0][0]                  
__________________________________________________________________________________________________
conv1d_27 (Conv1D)              (None, 256, 16)      4112        add_12[0][0]                     
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 256, 16)      0           conv1d_27[0][0]                  
__________________________________________________________________________________________________
lambda_13 (Lambda)              (None, 256, 16)      0           activation_13[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_13 (SpatialDr (None, 256, 16)      0           lambda_13[0][0]                  
__________________________________________________________________________________________________
conv1d_28 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_13[0][0]       
__________________________________________________________________________________________________
add_13 (Add)                    (None, 256, 16)      0           add_12[0][0]                     
                                                                 conv1d_28[0][0]                  
__________________________________________________________________________________________________
conv1d_29 (Conv1D)              (None, 256, 16)      4112        add_13[0][0]                     
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 256, 16)      0           conv1d_29[0][0]                  
__________________________________________________________________________________________________
lambda_14 (Lambda)              (None, 256, 16)      0           activation_14[0][0]              
__________________________________________________________________________________________________
spatial_dropout1d_14 (SpatialDr (None, 256, 16)      0           lambda_14[0][0]                  
__________________________________________________________________________________________________
conv1d_30 (Conv1D)              (None, 256, 16)      272         spatial_dropout1d_14[0][0]       
__________________________________________________________________________________________________
add_15 (Add)                    (None, 256, 16)      0           conv1d_2[0][0]                   
                                                                 conv1d_4[0][0]                   
                                                                 conv1d_6[0][0]                   
                                                                 conv1d_8[0][0]                   
                                                                 conv1d_10[0][0]                  
                                                                 conv1d_12[0][0]                  
                                                                 conv1d_14[0][0]                  
                                                                 conv1d_16[0][0]                  
                                                                 conv1d_18[0][0]                  
                                                                 conv1d_20[0][0]                  
                                                                 conv1d_22[0][0]                  
                                                                 conv1d_24[0][0]                  
                                                                 conv1d_26[0][0]                  
                                                                 conv1d_28[0][0]                  
                                                                 conv1d_30[0][0]                  
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 256, 16)      0           add_15[0][0]                     
__________________________________________________________________________________________________
dense (Dense)                   (None, 256, 3)       51          activation_15[0][0]              
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 256, 3)       0           dense[0][0]                      
==================================================================================================
Total params: 65,843
Trainable params: 65,843
Non-trainable params: 0
__________________________________________________________________________________________________
Epoch 1/4
 999/1000 [============================>.] - ETA: 0s - loss: 0.1099Epoch 1/4
 810/1000 [=======================>......] - ETA: 3s - loss: 0.1113
Epoch 00001: val_loss improved from inf to 0.11132, saving model to res/20201207_082948_model.h5
1000/1000 [==============================] - 159s 159ms/step - loss: 0.1100 - val_loss: 0.1113
Epoch 2/4
 999/1000 [============================>.] - ETA: 0s - loss: 0.0830Epoch 1/4
 811/1000 [=======================>......] - ETA: 2s - loss: 0.1283
Epoch 00002: val_loss did not improve from 0.11132
1000/1000 [==============================] - 154s 154ms/step - loss: 0.0831 - val_loss: 0.1285
Epoch 3/4
 999/1000 [============================>.] - ETA: 0s - loss: 0.0827Epoch 1/4
 811/1000 [=======================>......] - ETA: 3s - loss: 0.1090
Epoch 00003: val_loss improved from 0.11132 to 0.10919, saving model to res/20201207_082948_model.h5
1000/1000 [==============================] - 159s 159ms/step - loss: 0.0827 - val_loss: 0.1092
Epoch 4/4
 999/1000 [============================>.] - ETA: 0s - loss: 0.0814Epoch 1/4
 812/1000 [=======================>......] - ETA: 3s - loss: 0.1116
Epoch 00004: val_loss did not improve from 0.10919
1000/1000 [==============================] - 166s 166ms/step - loss: 0.0814 - val_loss: 0.1116
INFO:root:re-loading last best model
INFO:root:predicting
777/777 [==============================] - 14s 18ms/step
INFO:root:evaluating
INFO:root:[[3555131    6458   30969]
 [  12403   31631     274]
 [ 113317      55  228002]]
INFO:root:              precision    recall  f1-score   support

       noise      0.966     0.990     0.978   3592558
       pulse      0.829     0.714     0.767     44308
        sine      0.879     0.668     0.759    341374

    accuracy                          0.959   3978240
   macro avg      0.892     0.790     0.835   3978240
weighted avg      0.957     0.959     0.956   3978240

INFO:root:saving to res/20201207_082948_results.h5.

Training using command-line scripts

The training function das.train.train and all its arguments are also accessible from the command line via das train for use on the terminal. See here for a description of all command-line arguments. The command-line interface is generated with defopt.

For instance, training command above can be invoked from the command line:

das train --data-dir dat/dmel_single_raw.npy --save-dir res --model-name tcn --kernel-size 16 --nb-filters 16 --nb-hist 512 --nb-epoch 20 -i

Shell scripts are particularly useful if you want to fit a set of networks with with different configurations to optimize architecture. For instance, this scripts will fit networks with different numbers of TCN blocks (nb_conv) and filters (nb_filters):

#!/bin/bash
conda activate das

YSUFFIX="pulse"
MODELNAME='tcn'
DATADIR='../dat/dmel_single.npy'
SAVEDIR="res"

NB_HIST=2048
KERNEL_SIZE=32
NB_FILTERS=32
NB_CONV=3

for NB_CONV in 2 3 4
do
    for NB_FILTERS in 16 32 48 64
    do
        das train -i --nb-filters $NB_FILTERS --kernel-size $KERNEL_SIZE --nb-conv $NB_CONV --nb-hist $NB_HIST --save-dir $SAVEDIR --y-suffix $YSUFFIX --data-dir $DATADIR --model-name $MODELNAME
    done
done

A description of all command line arguments can be obtained by typing das train --help in a terminal:

!das train --help
usage: das train [-h] -d DATA_DIR [-y Y_SUFFIX] [--save-dir SAVE_DIR]
                 [--save-prefix SAVE_PREFIX] [-m MODEL_NAME]
                 [--nb-filters NB_FILTERS] [-k KERNEL_SIZE]
                 [--nb-conv NB_CONV] [-u [USE_SEPARABLE [USE_SEPARABLE ...]]]
                 [--nb-hist NB_HIST] [-i] [--no-ignore-boundaries]
                 [--batch-norm] [--no-batch-norm] [--nb-pre-conv NB_PRE_CONV]
                 [--pre-kernel-size PRE_KERNEL_SIZE]
                 [--pre-nb-filters PRE_NB_FILTERS] [--pre-nb-conv PRE_NB_CONV]
                 [-v VERBOSE] [--batch-size BATCH_SIZE] [--nb-epoch NB_EPOCH]
                 [--learning-rate LEARNING_RATE] [--reduce-lr]
                 [--no-reduce-lr] [--reduce-lr-patience REDUCE_LR_PATIENCE]
                 [-f FRACTION_DATA] [--seed SEED] [--batch-level-subsampling]
                 [--no-batch-level-subsampling] [-t] [--no-tensorboard]
                 [--log-messages] [--no-log-messages] [--nb-stacks NB_STACKS]
                 [-w] [--no-with-y-hist] [-x X_SUFFIX]

Train a DAS network.

optional arguments:
  -h, --help            show this help message and exit
  -d DATA_DIR, --data-dir DATA_DIR
                        Path to the directory or file with the dataset for training.
                        Accepts npy-dirs (recommended), h5 files or zarr files.
                        See documentation for how the dataset should be organized.
  -y Y_SUFFIX, --y-suffix Y_SUFFIX
                        Select training target by suffix.
                        Song-type specific targets can be created with a training dataset,
                        Defaults to '' (will use the standard target 'y')
  --save-dir SAVE_DIR   Directory to save training outputs.
                        The path of output files will constructed from the SAVE_DIR, an optional prefix, and the time stamp of the start of training.
                        Defaults to current directory ('./').
  --save-prefix SAVE_PREFIX
                        Prepend to timestamp.
                        Name of files created will be SAVE_DIR/SAVE_PREFIX + "_" + TIMESTAMP
                        or SAVE_DIR/ TIMESTAMP if SAVE_PREFIX is empty.
                        Defaults to '' (empty).
  -m MODEL_NAME, --model-name MODEL_NAME
                        Network architecture to use.
                        Use "tcn" (TCN) or "tcn_stft" (TCN with STFT frontend).
                        See das.models for a description of all models.
                        Defaults to 'tcn'.
  --nb-filters NB_FILTERS
                        Number of filters per layer.
                        Defaults to 16.
  -k KERNEL_SIZE, --kernel-size KERNEL_SIZE
                        Duration of the filters (=kernels) in samples.
                        Defaults to 16.
  --nb-conv NB_CONV     Number of TCN blocks in the network.
                        Defaults to 3.
  -u [USE_SEPARABLE [USE_SEPARABLE ...]], --use-separable [USE_SEPARABLE [USE_SEPARABLE ...]]
                        Specify which TCN blocks should use separable convolutions.
                        Provide as a space-separated sequence of "False" or "True.
                        For instance: "True False False" will set the first block in a
                        three-block (as given by nb_conv) network to use separable convolutions.
                        Defaults to False (no block uses separable convolution).
  --nb-hist NB_HIST     Number of samples processed at once by the network (a.k.a chunk size).
                        Defaults to 1024.
  -i, --ignore-boundaries
                        Minimize edge effects by discarding predictions at the edges of chunks.
                        Defaults to True.
  --no-ignore-boundaries
  --batch-norm          Batch normalize.
                        Defaults to True.
  --no-batch-norm
  --nb-pre-conv NB_PRE_CONV
                        Adds downsampling frontend.
                        TCN: adds a frontend of N conv blocks (conv-relu-batchnorm-maxpool2) to the TCN - useful for reducing the sampling rate for USV.
                        TCN_STFT: stft
                        Defaults to 0 (no frontend).
  --pre-kernel-size PRE_KERNEL_SIZE
                        [description]. Defaults to 3.
  --pre-nb-filters PRE_NB_FILTERS
                        [description]. Defaults to 16.
  --pre-nb-conv PRE_NB_CONV
                        [description]. Defaults to 3.
  -v VERBOSE, --verbose VERBOSE
                        Verbosity of training output (0 - no output(?), 1 - progress bar, 2 - one line per epoch).
                        Defaults to 2.
  --batch-size BATCH_SIZE
                        Batch size
                        Defaults to 32.
  --nb-epoch NB_EPOCH   Maximal number of training epochs.
                        Training will stop early if validation loss did not decrease in the last 20 epochs.
                        Defaults to 400.
  --learning-rate LEARNING_RATE
                        Learning rate of the model. Defaults should work in most cases.
                        Values typically range between 0.1 and 0.00001.
                        If None, uses per model defaults: "tcn" 0.0001, "tcn_stft" 0.0005).
                        Defaults to None.
  --reduce-lr           Reduce learning rate on plateau.
                        Defaults to False.
  --no-reduce-lr
  --reduce-lr-patience REDUCE_LR_PATIENCE
                        Number of epochs w/o a reduction in validation loss after which to trigger a reduction in learning rate.
                        Defaults to 5.
  -f FRACTION_DATA, --fraction-data FRACTION_DATA
                        Fraction of training and validation to use for training.
                        Defaults to 1.0.
  --seed SEED           Random seed to reproducible select fractions of the data.
                        Defaults to None (no seed).
  --batch-level-subsampling
                        Select fraction of data for training from random subset of shuffled batches.
                        If False, select a continuous chunk of the recording.
                        Defaults to False.
  --no-batch-level-subsampling
  -t, --tensorboard     Write tensorboard logs to save_dir.
                        Defaults to False.
  --no-tensorboard
  --log-messages        Sets logging level to INFO.
                        Defaults to False (will follow existing settings).
  --no-log-messages
  --nb-stacks NB_STACKS
                        Unused if model name is "tcn" or "tcn_stft". Defaults to 2.
  -w, --with-y-hist     Unused if model name is "tcn" or "tcn_stft". Defaults to True.
  --no-with-y-hist
  -x X_SUFFIX, --x-suffix X_SUFFIX
                        Select specific training data based on suffix (e.g. x_suffix).
                        Defaults to '' (will use the standard data 'x')