Search

class gdeep.search.Benchmark(models_dicts: List[Dict[str, Module | str]], dataloaders_dicts: List[Dict[str, List[DataLoader[Tuple[Tensor, Tensor]]] | str]], loss_fn: Callable[[Tensor, Tensor], Tensor], writer: SummaryWriter, training_metric: Callable[[Tensor, Tensor], float] | None = None, k_fold_class: BaseCrossValidator | None = None, regularizers_dicts: List[Mapping[str, Regularizer | None | str]] | None = None)

This is the generic class that allows the user to perform benchmarking over different datasets and models.

Args:

models_dicts :

each dictionary has two items, "name":"string" the name of the model and “model”:nn.Module a standard torch model

dataloaders_dicts :

each dictionary has two items, "name":"string" the name of the model and "dataloaders":list a list of standard torch.dataloaders, e.g. (dl_tr, dl_ts)

loss_fn:

loss function

writer:

tensorboard writer

training_metric:

the function that computes the metric: it shall have two arguments, one for the prediction and the other for the ground truth

k_fold_class:

the class instance to implement the KFold, can be any of the Splitter classes of sklearn. More info at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection

Examples:

from torch import nn
from torch.optim import SGD
from torch.utils.data.sampler import SubsetRandomSampler
from gdeep.data.datasets import DatasetBuilder, DataLoaderBuilder
from gdeep.data.preprocessors import ToTensorImage
from gdeep.search import Benchmark
from gdeep.models import FFNet
# initialise dataloader list of dictionaries
dataloaders_dicts = []
bd = DatasetBuilder(name="CIFAR10")
# build first dataloader
ds_tr, *_ = bd.build()
transformation = ToTensorImage((32, 32))
transformed_ds_tr = transformation.attach_transform_to_dataset(ds_tr)
# cut down some data
test_indices = [64 * 5 + x for x in range(32 * 3)]
train_indices = [x for x in range(32 * 2)]
dl = DataLoaderBuilder((transformed_ds_tr, transformed_ds_tr))
dl_tr, dl_val, _ = dl.build(
    (
        {"batch_size": 32, "sampler": SubsetRandomSampler(train_indices)},
        {"batch_size": 32, "sampler": SubsetRandomSampler(test_indices)},
    )
)
# prepare the dataloader dictionary
temp_dict = {}
temp_dict["name"] = "CIFAR10_1000"
temp_dict["dataloaders"] = (dl_tr, dl_val, _)
# store the dictionary to the list
dataloaders_dicts.append(temp_dict)
# repeat with another dataset
db = DatasetBuilder(name="DoubleTori")
ds_tr, ds_val, _ = db.build()
dl_tr, dl_ts, _ = DataLoaderBuilder((ds_tr, ds_val)).build(
    ({"batch_size": 48}, {"batch_size": 32})
)
temp_dict = {}
temp_dict["name"] = "double_tori"
temp_dict["dataloaders"] = (dl_tr, dl_ts)
dataloaders_dicts.append(temp_dict)
# prepare the list of model dictionaries
models_dicts = []
# define your model as a torch.nn.Module
model = model2()
temp_dict = {}
temp_dict["name"] = "resnet18"
temp_dict["model"] = model
models_dicts.append(temp_dict)
# avoid having exposed paramters that wll not be searched on
class model_no_param(nn.Module):
    def __init__(self):
        super(model_no_param, self).__init__()
        self.mod = FFNet([3, 5, 5, 2])

    def forward(self, x):
        return self.mod(x)
# initialise the dictionary of the other model
model5 = model_no_param()
temp_dict = {}
temp_dict["name"] = "ffnn"
temp_dict["model"] = model5
# append to the model list of dictionaries
models_dicts.append(temp_dict)
# standard pytorch loss
loss_fn = nn.CrossEntropyLoss()

# initialise benchmark
bench = Benchmark(
    models_dicts, dataloaders_dicts, loss_fn, writer, k_fold_class=KFold(3)
)
# start the benchmarking
bench.start(SGD, 1, True, {"lr": 0.01}, {"batch_size": 23})

start(optimizer: Type[Optimizer], n_epochs: int = 10, cross_validation: bool = False, optimizers_param: Dict[str, Any] | None = None, dataloaders_param: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, scheduler_params: Dict[str, Any] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '') → None

Method to be called when starting the benchmarking

Args:

optimizer:

a torch optimizers class (not the instance)

n_epochs:

number of training epochs

cross_validation:

whether or not to use cross-validation

optimizers_param:

dictionary of the optimizers parameters, e.g. {"lr": 0.001}

dataloaders_param:

dictionary of the dataloaders parameters, e.g. {"batch_size": 32}

lr_scheduler:

a learning rate scheduler class (not instance)

scheduler_params:

learning rate scheduler parameters

profiling:

whether or not you want to activate the profiler

parallel_tpu:

boolean value to run the computations on multiple TPUs

keep_training:

This flag allows to restart a training from the existing optimizer as well as the existing model

store_grad_layer_hist:

This flag allows to store the gradients and the layer values in tensorboard for each epoch

n_accumulated_grads:

number of accumulated gradients. It is considered only if given a positive integer

writer_tag:

string to be added to the tensorboard items title

class gdeep.search.GiottoSummaryWriter(log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix='')

add_hparams(hparam_dict: Dict[str, Any], metric_dict: Dict[str, Any], hparam_domain_discrete: Dict[str, List[Any]] | None = None, run_name: None | str = None, scalars_lists: List[List[Tuple[Any, int]]] | None = None, best_not_last: bool = False)

Add a set of hyperparameters to be compared in TensorBoard. Args:

hparam_dict (dict):

Each key-value pair in the dictionary is the name of the hyper parameter and it’s corresponding value. The type of the value can be one of bool, string, float, int, or None.

metric_dict :

Each key-value pair in the dictionary is the name of the metric and it’s corresponding value. Note that the key used here should be unique in the tensorboard record. Otherwise the value you added by add_scalar will be displayed in hparam plugin. In most cases, this is unwanted.

hparam_domain_discrete:

A dictionary that contains names of the hyperparameters and all discrete values they can hold

run_name :

Name of the run, to be included as part of the logdir. If unspecified, will use current timestamp.

scalars_lists :

The lists for the loss and accuracy plots. This is a list with two lists (one for accuracy and one for the loss). Each one of the inner lists contain the pairs (metric_value, epoch).

best_not_last:

boolean flag to decide what value to store in the tensorboard tables for the whole training cycle

Examples::

from torch.utils.tensorboard import SummaryWriter with GiottoSummaryWriter() as w:

for i in range(5):

w.add_hparams({‘lr’: 0.1*i, ‘bsize’: i},

{‘hparam/accuracy’: 10*i, ‘hparam/loss’: 10*i})

class gdeep.search.HPOConfig(optimizers: List[Type[Optimizer]], n_epochs: int = 1, cross_validation: bool = False, optimizers_params: Dict[str, Any] | None = None, dataloaders_params: Dict[str, Any] | None = None, models_hyperparams: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, schedulers_params: Dict[str, Any] | None = None, regularization_params: Dict[str, Any] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '')

Config class for thee HPO parameters

Args:

optimizers:

list of torch optimizers classes, not instances

n_epochs:

number of training epochs

cross_validation:

whether or not to use cross-validation

optimizers_params:

dictionary of optimizers params

dataloaders_params:

dictionary of dataloaders parameters

models_hyperparams:

dictionary of model parameters

lr_scheduler:

torch learning rate scheduler class

schedulers_params:

learning rate scheduler parameters

profiling :

whether or not you want to activate the profiler

parallel_tpu:

boolean value to run the computations on multiple TPUs

n_accumulated_grads (int, default=0):

number of accumulated gradients. It is considered only if a positive integer

keep_training:

bool flag to decide whether to continue training or not

store_grad_layer_hist:

flag to store the gradients of the layers in the tensorboard histograms

writer_tag:

tag to prepend to the output on tensorboard

to_dict() → Dict[str, Any]

method to transform the config file into a dictionary

class gdeep.search.HyperParameterOptimization(obj: Trainer | Benchmark, search_metric: typing_extensions.Literal[loss, accuracy] = 'loss', n_trials: int = 10, best_not_last: bool = False, pruner: BasePruner | None = None, sampler=None, db_url: None | str = None, study_name: None | str = None)

This is the generic class that allows the user to perform hyperparameter search over several parameters such as learning rate, optimizer. Args:

obj :

either a Trainer or a Benchmark class instance

search_metric :

either 'loss' or 'accuracy'

n_trials :

number of total search trials

best_not_last :

boolean flag that is True would use the best metric over epochs averaged over the folds in CV rather than the last value of the metrics over the epochs averaged over the folds

pruner :

Instance of an optuna pruner, can be user-defined

sampler :

If left unspecified, TPESample is used during single-objective optimization and NSGAIISampler during multi-objective optimization

db_url:

name of the database to connect to. For example mysql+mysqldb://usr:psw@host:port/db_name

study_name:

name of the optuna study

Examples:

from gdeep.search import HyperParameterOptimization
# initialise hpo, you need a `trainer`!
search = HyperParameterOptimization(trainer, "accuracy", 2, best_not_last=True)
# if you want to store pickle files of the models instead of the state_dicts
search.store_pickle = True
# dictionaries of hyperparameters
optimizers_params = {"lr": [0.001, 0.01]}
dataloaders_params = {"batch_size": [32, 64, 16]}
models_hyperparams = {"n_nodes": ["200"]}
# starting the HPO
search.start(
    (SGD, Adam),
    3,
    False,
    optimizers_params,
    dataloaders_params,
    models_hyperparams,
    n_accumulated_grads=2,
)

start(optimizers: List[Type[Optimizer]], n_epochs: int = 1, cross_validation: bool = False, optimizers_params: Dict[str, Any] | None = None, dataloaders_params: Dict[str, Any] | None = None, models_hyperparams: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, schedulers_params: Dict[str, Any] | None = None, regularization_params: Dict[str, Any] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '') → None

method to be called when starting the hyperparameter optimization Args:

optimizers:

list of torch optimizers classes, not instances

n_epochs:

number of training epochs

cross_validation:

whether or not to use cross-validation

optimizers_params:

dictionary of optimizers params

dataloaders_params:

dictionary of dataloaders parameters

models_hyperparams:

dictionary of model parameters

lr_scheduler:

torch learning rate scheduler class

schedulers_params:

learning rate scheduler parameters

profiling :

whether or not you want to activate the profiler

parallel_tpu:

boolean value to run the computations on multiple TPUs

n_accumulated_grads (int, default=0):

number of accumulated gradients. It is considered only if a positive integer

keep_training:

bool flag to decide whether to continue training or not

store_grad_layer_hist:

flag to store the gradients of the layers in the tensorboard histograms

writer_tag:

tag to prepend to the output on tensorboard

class gdeep.search.TrainerConfig(optimizer: Type[Optimizer], n_epochs: int = 10, cross_validation: bool = False, optimizers_param: Dict[str, Any] | None = None, dataloaders_param: Dict[str, Any] | None = None, lr_scheduler: Type[_LRScheduler] | None = None, scheduler_params: Dict[str, Any] | None = None, optuna_params: Tuple[BaseTrial, str] | None = None, profiling: bool = False, parallel_tpu: bool = False, keep_training: bool = False, store_grad_layer_hist: bool = False, n_accumulated_grads: int = 0, writer_tag: str = '')

Configuration class that contains the parameters of the Trainer and of the Benchmark class

Args:

optimizer:

optimizer class, not the instance

n_epochs:

number of training epochs

optimizers_params:

dictionary of the optimizers parameters, e.g. {“lr”: 0.001}

dataloaders_params:

dictionary of the dataloaders parameters

models_hyperparams:

dictionary of the model parameters

lr_scheduler:

a learning rate scheduler class

schedulers_params:

learning rate scheduler parameters

optuna_params:

a tuple with the optuna trial and the search metric (a string).

profiling:

whether or not you want to activate the profiler

parallel_tpu:

boolean value to run the computations on multiple TPUs

keep_training:

This flag allows to restart a training from the existing optimizer as well as the existing model

store_grad_layer_hist:

This flag allows to store the gradients and the layer values in tensorboard for each epoch

n_accumulated_grads:

this is the number of accumulated grads. It is taken into account only for positive integers

writer_tag:

tag to prepend to the output on tensorboard

to_dict() → Dict[str, Any]

method to transform the config file into a dictionary

gdeep.search.clean_up_files(fun)

Decorator to remove all the files created by the method it decorates