Trainer

class gdeep.trainer.Regularizer(*args, **kwargs)

class gdeep.trainer.TihonovRegularizer(lamda: float, p: int)

An example implementation of a p-norm regularizer This regularizer penalizes the p-norm (to the pth power) of the network weights. For example, p=1 is LASSO and p=2 is the Ridge Regression. reg_params: p (int), (Optional)

regularization_penalty(model: Module) → Tensor

The regularization penalty is sum of the pth powers of the p-norms of the regression coefficients, i.e. network weights

class gdeep.trainer.TopologicalRegularizer(lamda, data: TopologicalRegularizerData)

A topological regularizer for a 2-class classifier

regularization_penalty(model: Module) → Tensor

The penalty is the squared sum of the function values at the critical points.

class gdeep.trainer.TopologicalRegularizerData(regularization_dataset: Tensor, n_neighbors: Tuple[int, int] = (6, 20), hom_dim: Tuple[int, ...] = (0, 1, 2, 3, 4, 5, 6), expansion: int = 6)

Container for the data needed for TopologicalRegularizer. Args:

regularization_dataset:

tensor describing where the linear surrogate function is evaluated

n_neighbors:

sizes of the KNN neighbors used for computing the local homology

hom_dim:

the homologies to be computed

expansion:

the size of the clique-expansion when computing homology

class gdeep.trainer.Trainer(model: Module, dataloaders: List[DataLoader[Tuple[Tensor | List[Tensor], Tensor]]], loss_fn: Callable[[Tensor, Tensor], Tensor], writer: SummaryWriter | None = None, training_metric: Callable[[Tensor, Tensor], float] | None = None, k_fold_class: BaseCrossValidator | None = None, print_every: int = 1, regularizer: Regularizer | None = None)

This is the generic class that allows the user to benchmark models over architectures datasets, regularisation, metrics… in one line of code.

Args:

model :

standard torch model

dataloaders (list of utils.DataLoader):

list of standard torch DataLoaders, e.g. [dl_tr, dl_val, dl_ts]

loss_fn :

loss function to average over batches

writer :

tensorboard writer

training_metric:

the function that computes the metric: it shall have two arguments, one for the prediction and the other for the ground truth

k_fold_class (sklearn.model_selection, default KFold(5, shuffle=True)):

the class instance to implement the KFold, can be any of the Splitter classes of sklearn. More info at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection

print_every:

The number of training steps performed between each information printout

regularizer:

a gdeep regularizer

Examples:

from torch import nn
from torch.optim import SGD
from sklearn.model_selection import StratifiedKFold
from gdeep.models import FFNet
from gdeep.trainer import Trainer
from gdeep.data import BuildDatasets, BuildDataLoaders
from gdeep.search import GiottoSummaryWriter
# model
class model1(nn.Module):
    def __init__(self):
        super(model1, self).__init__()
        self.seqmodel = nn.Sequential(nn.Flatten(), FFNet(arch=[3, 5, 10, 5, 2]))
    def forward(self, x):
        return self.seqmodel(x)

model = model1()
# dataloaders
bd = BuildDatasets(name="DoubleTori")
ds_tr, ds_val, _ = bd.build_datasets()
dl = BuildDataLoaders((ds_tr, ds_val))
dl_tr, dl_val, dl_ts = dl.build_dataloaders(batch_size=23)

# loss function
loss_fn = nn.CrossEntropyLoss()
# tb writer
writer = GiottoSummaryWriter()
# pipeline
pipe = Trainer(model, [dl_tr, dl_val, dl_ts],
                loss_fn, writer, None,
                StratifiedKFold(5, shuffle=True))
# then one needs to train the model using the pipeline!
pipe.train(SGD, 2, True, {"lr": 0.001}, n_accumulated_grads=5)

static copy_dataloader_params(original_dataloader: DataLoader[Tuple[Tensor | List[Tensor], Tensor]]) → Dict[str, Any]

returns the dict of init parameters

evaluate_classification(num_class: int | None = None, dl: DataLoader[Tuple[Tensor | List[Tensor], Tensor]] | None = None) → Tuple[float, float, ndarray]

Method to evaluate the performance of the model.

Args:

num_class:

number of classes

dl :

the Dataloader to evaluate. If None, we use the training dataloader in self

Returns:

(float, float, 2darray):

the accuracy, loss and confusion matrix.

parallel_tpu_training_loops(n_epochs: int, dl_tr_old: DataLoader[Tuple[Tensor | List[Tensor], Tensor]], dl_val_old: DataLoader[Tuple[Tensor | List[Tensor], Tensor]], optimizers_param: Dict[str, Any], lr_scheduler: Type[_LRScheduler] | None = None, scheduler: _LRScheduler | None = None, check_optuna: bool = False, search_metric: None | str = None, trial: BaseTrial | None = None, cross_validation: bool = False) → Tuple[float, float]

Experimental function to run all the training cycles on colab TPUs in parallel. Note: cross_validation parameter as well as profiling are ignored.

Args:

n_epochs:

number of training epochs

dl_tr_old:

training dataloader

dl_val_old:

validation dataloader parameters, e.g. {‘batch_size’: 32}

optimizers_param:

dictionary of parameters for the optimizers

lr_scheduler :

a learning rate scheduler class

scheduler:

the actual scheduler instance

check_optuna :

boolean to store the optuna results of the trial

search_metric:

either 'loss' or 'accuracy', this corresponds to the gridsearch criterion

trial:

the optuna trial

cross_validation :

the boolean flag for cross validation

Returns:

(float, float):

the validation loss and validation accuracy

register_pipe_hook(callable: Callable[[int, Optimizer, ModelExtractor, SummaryWriter | None], Any]) → None

This method registers a function that will be called after each trainign step.

The arguments of the callable function are, in this order:

• current epoch number

• current optimizer instance

• the ModelExtractor instance at that epoch

• the tensorboard writer

Args:

callable (Callable):

the function to register

train(optimizer: Type[Optimizer], n_epochs: int = 10, cross_validation: bool = False, optimizers_param: Dict[str, Any] | None = None, dataloaders_param: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, scheduler_params: Dict[str, Any] | None = None, optuna_params: Tuple[BaseTrial, str] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '') → Tuple[float, float]

Function to run all the training cycles.

Args:

optimizer:

the torch optimiser class, like SGD

n_epochs :

number of training epochs

cross_validation:

whether or not to perform five-fold cross-validation

dataloaders_param:

dictionary of the dataloaders parameters, e.g. {‘batch_size’: 32}. If None, then the parameters of the training and validation dataloaders will be used.

optimizers_param:

dictionary of the optimizers parameters, e.g. {“lr”: 0.001}

lr_scheduler:

a learning rate scheduler class

scheduler_params:

learning rate scheduler parameters

optuna_params :

the parameters (trial, search_metric) used in the gridsearch. Safely ignore for standard trainings

profiling:

whether or not you want to activate the profiler

parallel_tpu:

Use or not parallel TPU cores. Still experimental!

keep_training:

This flag allows to restart a training from the existing optimizer as well as the existing model

store_grad_layer_hist:

This flag allows to store the gradients and the layer values in tensorboard for each epoch

n_accumulated_grads:

this is the number of accumulated gradients. Only a positive number will be taken into account

writer_tag:

the tensorboard writer tag

Returns:

(float, float):

the validation loss and accuracy if there is cross validation, the validation data loader is ignored. On the other hand, if there cross_validation = False then the test loss and accuracy is returned.

class gdeep.trainer.TrainerConfig(optimizer: Type[Optimizer], n_epochs: int = 10, cross_validation: bool = False, optimizers_param: Dict[str, Any] | None = None, dataloaders_param: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, scheduler_params: Dict[str, Any] | None = None, optuna_params: Tuple[BaseTrial, str] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '')

Configuration class that contains the parameters of the Trainer and of the Benchmark class

Args:

optimizer:

optimizer class, not the instance

n_epochs:

number of training epochs

optimizers_params:

dictionary of the optimizers parameters, e.g. {“lr”: 0.001}

dataloaders_params:

dictionary of the dataloaders parameters

models_hyperparams:

dictionary of the model parameters

lr_scheduler:

a learning rate scheduler class

schedulers_params:

learning rate scheduler parameters

optuna_params:

a tuple with the optuna trial and the search metric (a string).

profiling:

whether or not you want to activate the profiler

parallel_tpu:

boolean value to run the computations on multiple TPUs

keep_training:

This flag allows to restart a training from the existing optimizer as well as the existing model

store_grad_layer_hist:

This flag allows to store the gradients and the layer values in tensorboard for each epoch

n_accumulated_grads:

this is the number of accumulated grads. It is taken into account only for positive integers

writer_tag:

tag to prepend to the output on tensorboard

to_dict() → Dict[str, Any]

method to transform the config file into a dictionary

gdeep.trainer.accuracy(prediction: Tensor, y: Tensor) → float

This function computes the accuracy for the given prediction and expected output

Args:

prediction:

the output of your model. If X is a tensor, then prediction = model(X)

y:

the corresponding expected results

Returns:

float:

the value of the accuracy