DL

Core

class catalyst.dl.core.callback.MeterMetricsCallback(metric_names: List[str], meter_list: List, input_key: str = 'targets', output_key: str = 'logits', class_names: List[str] = None, num_classes: int = 2, activation: str = 'Sigmoid')

Bases: catalyst.core.callback.Callback

A callback that tracks metrics through meters and prints metrics for each class on state.on_loader_end. This callback works for both single metric and multi-metric meters.

__init__(metric_names: List[str], meter_list: List, input_key: str = 'targets', output_key: str = 'logits', class_names: List[str] = None, num_classes: int = 2, activation: str = 'Sigmoid')

Parameters

• metric_names (List[str]) – of metrics to print Make sure that they are in the same order that metrics are outputted by the meters in meter_list

• meter_list (list-like) – List of meters.meter.Meter instances len(meter_list) == num_classes

• input_key (str) – input key to use for metric calculation specifies our y_true.

• output_key (str) – output key to use for metric calculation; specifies our y_pred

• class_names (List[str]) – class names to display in the logs. If None, defaults to indices for each class, starting from 0.

• num_classes (int) – Number of classes; must be > 1

• activation (str) – An torch.nn activation applied to the logits. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

on_batch_end(state: catalyst.dl.core.state.State)

on_loader_end(state: catalyst.dl.core.state.State)

on_loader_start(state)

class catalyst.dl.core.experiment.Experiment

Bases: catalyst.core.experiment._Experiment

get_native_batch(stage: str, loader: Union[str, int] = 0, data_index: int = 0)

Returns a batch from experiment loader

Parameters

• stage (str) – stage name

• loader (Union[str, int]) – loader name or its index, default is the first loader

• data_index (int) – index in dataset from the loader

class catalyst.dl.core.runner.Runner(model: torch.nn.modules.module.Module = None, device: Union[str, torch.device] = None)

Bases: catalyst.core.runner._Runner

Deep Learning Runner for different supervised, unsupervised, gan, etc runs

class catalyst.dl.core.state.State(*, device: Union[str, torch.device] = None, model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, criterion: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, optimizer: Union[torch.optim.optimizer.Optimizer, Dict[str, torch.optim.optimizer.Optimizer]] = None, scheduler: Union[torch.optim.lr_scheduler._LRScheduler, Dict[str, torch.optim.lr_scheduler._LRScheduler]] = None, callbacks: Dict[str, Callback] = None, logdir: str = None, stage: str = 'infer', num_epochs: int = None, main_metric: str = 'loss', minimize_metric: bool = True, valid_loader: str = 'valid', checkpoint_data: Dict = None, is_check_run: bool = False, **kwargs)

Bases: catalyst.core.state._State

Callbacks

class catalyst.dl.callbacks.gan.WassersteinDistanceCallback(prefix: str = 'wasserstein_distance', real_validity_output_key: str = 'real_validity', fake_validity_output_key: str = 'fake_validity', multiplier: float = 1.0)

Bases: catalyst.core.callbacks.metrics.MetricCallback

Callback to compute Wasserstein distance metric

__init__(prefix: str = 'wasserstein_distance', real_validity_output_key: str = 'real_validity', fake_validity_output_key: str = 'fake_validity', multiplier: float = 1.0)

Parameters

• prefix –

• real_validity_output_key –

• fake_validity_output_key –

get_wasserstein_distance(real_validity, fake_validity)

Computes Wasserstein distance

class catalyst.dl.callbacks.gan.GradientPenaltyCallback(real_input_key: str = 'data', fake_output_key: str = 'fake_data', condition_keys: List[str] = None, critic_model_key: str = 'critic', critic_criterion_key: str = 'critic', real_data_criterion_key: str = 'real_data', fake_data_criterion_key: str = 'fake_data', condition_args_criterion_key: str = 'critic_condition_args', prefix: str = 'loss', criterion_key: str = None, multiplier: float = 1.0)

Bases: catalyst.core.callbacks.criterion.CriterionCallback

Criterion Callback to compute Gradient Penalty

__init__(real_input_key: str = 'data', fake_output_key: str = 'fake_data', condition_keys: List[str] = None, critic_model_key: str = 'critic', critic_criterion_key: str = 'critic', real_data_criterion_key: str = 'real_data', fake_data_criterion_key: str = 'fake_data', condition_args_criterion_key: str = 'critic_condition_args', prefix: str = 'loss', criterion_key: str = None, multiplier: float = 1.0)

Parameters

• real_input_key – real data key in state.input

• fake_output_key – fake data key in state.output

• condition_keys – all condition keys in state.input for critic

• critic_model_key – key for critic model in state.model

• critic_criterion_key – key for critic model in criterion

• real_data_criterion_key – key for real data in criterion

• fake_data_criterion_key – key for fake data in criterion

• condition_args_criterion_key – key for all condition args in criterion

• prefix –

• criterion_key –

• multiplier –

class catalyst.dl.callbacks.gan.WeightClampingOptimizerCallback(grad_clip_params: Dict = None, accumulation_steps: int = 1, optimizer_key: str = None, loss_key: str = 'loss', decouple_weight_decay: bool = True, weight_clamp_value: float = 0.01)

Bases: catalyst.core.callbacks.optimizer.OptimizerCallback

Optimizer callback + weights clipping after step is finished

__init__(grad_clip_params: Dict = None, accumulation_steps: int = 1, optimizer_key: str = None, loss_key: str = 'loss', decouple_weight_decay: bool = True, weight_clamp_value: float = 0.01)

Parameters

• grad_clip_params –

• accumulation_steps –

• optimizer_key –

• loss_key –

• decouple_weight_decay –

• weight_clamp_value – value to clamp weights after each optimization iteration Attention: will clamp WEIGHTS, not GRADIENTS

on_batch_end(state: catalyst.dl.core.state.State)

On batch end event

class catalyst.dl.callbacks.inference.InferCallback(out_dir=None, out_prefix=None)

Bases: catalyst.core.callback.Callback

on_batch_end(state: catalyst.dl.core.state.State)

on_loader_end(state: catalyst.dl.core.state.State)

on_loader_start(state: catalyst.dl.core.state.State)

on_stage_start(state: catalyst.dl.core.state.State)

class catalyst.dl.callbacks.inference.InferMaskCallback(out_dir=None, out_prefix=None, input_key=None, output_key=None, name_key=None, mean=None, std=None, threshold: float = 0.5, mask_strength: float = 0.5, mask_type: str = 'soft')

Bases: catalyst.core.callback.Callback

on_batch_end(state: catalyst.dl.core.state.State)

on_loader_start(state: catalyst.dl.core.state.State)

on_stage_start(state: catalyst.dl.core.state.State)

class catalyst.dl.callbacks.mixup.MixupCallback(input_key: str = 'targets', output_key: str = 'logits', fields: List[str] = ('features', ), alpha=1.0, on_train_only=True, **kwargs)

Bases: catalyst.core.callbacks.criterion.CriterionCallback

Callback to do mixup augmentation.

Paper: https://arxiv.org/abs/1710.09412

Note

MixupCallback is inherited from CriterionCallback and does its work.

You may not use them together.

__init__(input_key: str = 'targets', output_key: str = 'logits', fields: List[str] = ('features', ), alpha=1.0, on_train_only=True, **kwargs)

Parameters

• fields (List[str]) – list of features which must be affected.

• alpha (float) – beta distribution a=b parameters. Must be >=0. The more alpha closer to zero the less effect of the mixup.

• on_train_only (bool) – Apply to train only. As the mixup use the proxy inputs, the targets are also proxy. We are not interested in them, are we? So, if on_train_only is True, use a standard output/metric for validation.

on_batch_start(state: catalyst.dl.core.state.State)

on_loader_start(state: catalyst.dl.core.state.State)

class catalyst.dl.callbacks.scheduler.LRFinder(final_lr, scale='log', num_steps=None, optimizer_key=None)

Bases: catalyst.core.callbacks.scheduler.LRUpdater

Helps you find an optimal learning rate for a model, as per suggestion of 2015 CLR paper. Learning rate is increased in linear or log scale, depending on user input.

https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html

__init__(final_lr, scale='log', num_steps=None, optimizer_key=None)

Parameters

• final_lr – final learning rate to try with

• scale – learning rate increasing scale (“log” or “linear”)

• num_steps – number of batches to try; if None - whole loader would be used.

• optimizer_key – which optimizer key to use for learning rate scheduling

calc_lr()

on_batch_end(state: catalyst.dl.core.state.State)

on_loader_start(state: catalyst.dl.core.state.State)

Metrics

class catalyst.dl.callbacks.metrics.accuracy.AccuracyCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'accuracy', accuracy_args: List[int] = None, num_classes: int = None, threshold: float = None, activation: str = None)

Bases: catalyst.core.callbacks.metrics.MultiMetricCallback

Accuracy metric callback.

It can be used either for

• multi-class task:

  -you can use accuracy_args. -threshold and activation are not required. -input_key point on tensor: batch_size. -output_key point on tensor: batch_size x num_classes.

• OR multi-label task, in this case:

  -you must specify threshold and activation. -accuracy_args and num_classes will not be used (because of there is no method to apply top-k in multi-label classification). -input_key, output_key point on tensor: batch_size x num_classes. -output_key point on a tensor with binary vectors.

There is no need to choose a type (multi-class/multi label). An appropriate type will be chosen automatically via shape of tensors.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'accuracy', accuracy_args: List[int] = None, num_classes: int = None, threshold: float = None, activation: str = None)

Parameters

• input_key (str) – input key to use for accuracy calculation; specifies our y_true.

• output_key (str) – output key to use for accuracy calculation; specifies our y_pred.

• prefix (str) – key for the metric’s name

• accuracy_args (List[int]) – specifies which accuracy@K to log. [1] - accuracy [1, 3] - accuracy at 1 and 3 [1, 3, 5] - accuracy at 1, 3 and 5

• num_classes (int) – number of classes to calculate accuracy_args if accuracy_args is None

• threshold (float) – threshold for outputs binarization.

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax”].

class catalyst.dl.callbacks.metrics.accuracy.MapKCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'map', map_args: List[int] = None, num_classes: int = None)

Bases: catalyst.core.callbacks.metrics.MultiMetricCallback

mAP@k metric callback.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'map', map_args: List[int] = None, num_classes: int = None)

Parameters

• input_key (str) – input key to use for calculation mean average accuracy at k; specifies our y_true.

• output_key (str) – output key to use for calculation mean average accuracy at k; specifies our y_pred.

• prefix (str) – key for the metric’s name

• map_args (List[int]) – specifies which map@K to log. [1] - map@1 [1, 3] - map@1 and map@3 [1, 3, 5] - map@1, map@3 and map@5

• num_classes (int) – number of classes to calculate map_args if map_args is None

class catalyst.dl.callbacks.metrics.auc.AUCCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'auc', class_names: List[str] = None, num_classes: int = 2, activation: str = 'Sigmoid')

Bases: catalyst.dl.core.callback.MeterMetricsCallback

Calculates the AUC per class for each loader. Currently, supports binary and multi-label cases.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'auc', class_names: List[str] = None, num_classes: int = 2, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for auc calculation specifies our y_true.

• output_key (str) – output key to use for auc calculation; specifies our y_pred

• prefix (str) – name to display for auc when printing

• class_names (List[str]) – class names to display in the logs. If None, defaults to indices for each class, starting from 0.

• num_classes (int) – Number of classes; must be > 1

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

class catalyst.dl.callbacks.metrics.dice.DiceCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'dice', eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: catalyst.core.callbacks.metrics.MetricCallback

Dice metric callback.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'dice', eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for dice calculation; specifies our y_true.

• output_key (str) – output key to use for dice calculation; specifies our y_pred.

class catalyst.dl.callbacks.metrics.dice.MulticlassDiceMetricCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'dice', class_names=None)

Bases: catalyst.core.callback.Callback

Global Multi-Class Dice Metric Callback: calculates the exact dice score across multiple batches. This callback is good for getting the dice score with small batch sizes where the batchwise dice is noisier.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'dice', class_names=None)

Parameters

• input_key (str) – input key to use for dice calculation; specifies our y_true.

• output_key (str) – output key to use for dice calculation; specifies our y_pred.

• prefix (str) – prefix for printing the metric

• class_names (dict/List) – if dictionary, should be: {class_id: class_name, …} where class_id is an integer This allows you to ignore class indices. if list, make sure it corresponds to the number of classes

on_batch_end(state: catalyst.dl.core.state.State)

Records the confusion matrix at the end of each batch.

on_loader_end(state: catalyst.dl.core.state.State)

class catalyst.dl.callbacks.metrics.f1_score.F1ScoreCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'f1_score', beta: float = 1.0, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: catalyst.core.callbacks.metrics.MetricCallback

F1 score metric callback.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'f1_score', beta: float = 1.0, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for iou calculation specifies our y_true.

• output_key (str) – output key to use for iou calculation; specifies our y_pred

• prefix (str) – key to store in logs

• beta (float) – beta param for f_score

• eps (float) – epsilon to avoid zero division

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

class catalyst.dl.callbacks.metrics.iou.IouCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'iou', eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: catalyst.core.callbacks.metrics.MetricCallback

IoU (Jaccard) metric callback.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'iou', eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for iou calculation specifies our y_true.

• output_key (str) – output key to use for iou calculation; specifies our y_pred

• prefix (str) – key to store in logs

• eps (float) – epsilon to avoid zero division

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

catalyst.dl.callbacks.metrics.iou.JaccardCallback

alias of catalyst.dl.callbacks.metrics.iou.IouCallback

class catalyst.dl.callbacks.metrics.iou.ClasswiseIouCallback(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'iou', classes: List[str] = None, num_classes: int = None, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: catalyst.core.callbacks.metrics.MultiMetricCallback

Classwise IoU (Jaccard) metric callback.

__init__(input_key: str = 'targets', output_key: str = 'logits', prefix: str = 'iou', classes: List[str] = None, num_classes: int = None, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for iou calculation specifies our y_true.

• output_key (str) – output key to use for iou calculation; specifies our y_pred

• prefix (str) – key to store in logs (will be prefix_class_name)

• classes (List[str]) – list of class names You should specify either ‘classes’ or ‘num_classes’

• num_classes (int) – number of classes You should specify either ‘classes’ or ‘num_classes’

• eps (float) – epsilon to avoid zero division

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

catalyst.dl.callbacks.metrics.iou.ClasswiseJaccardCallback

alias of catalyst.dl.callbacks.metrics.iou.ClasswiseIouCallback

class catalyst.dl.callbacks.metrics.ppv_tpr_f1.PrecisionRecallF1ScoreCallback(input_key: str = 'targets', output_key: str = 'logits', class_names: List[str] = None, num_classes: int = 2, threshold: float = 0.5, activation: str = 'Sigmoid')

Bases: catalyst.dl.core.callback.MeterMetricsCallback

Calculates the global precision (positive predictive value or ppv), recall (true positive rate or tpr), and F1-score per class for each loader. Currently, supports binary and multi-label cases.

__init__(input_key: str = 'targets', output_key: str = 'logits', class_names: List[str] = None, num_classes: int = 2, threshold: float = 0.5, activation: str = 'Sigmoid')

Parameters

• input_key (str) – input key to use for metric calculation specifies our y_true.

• output_key (str) – output key to use for metric calculation; specifies our y_pred

• class_names (List[str]) – class names to display in the logs. If None, defaults to indices for each class, starting from 0.

• num_classes (int) – Number of classes; must be > 1

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [‘none’, ‘Sigmoid’, ‘Softmax2d’]

Experiment

class catalyst.dl.experiment.base.BaseExperiment(model: torch.nn.modules.module.Module, loaders: OrderedDict[str, DataLoader], callbacks: Union[OrderedDict[str, Callback], List[Callback]] = None, logdir: str = None, stage: str = 'train', criterion: torch.nn.modules.module.Module = None, optimizer: torch.optim.optimizer.Optimizer = None, scheduler: torch.optim.lr_scheduler._LRScheduler = None, num_epochs: int = 1, valid_loader: str = 'valid', main_metric: str = 'loss', minimize_metric: bool = True, verbose: bool = False, check_run: bool = False, state_kwargs: Dict = None, checkpoint_data: Dict = None, distributed_params: Dict = None, monitoring_params: Dict = None, initial_seed: int = 42)

Bases: catalyst.dl.core.experiment.Experiment

Super-simple one-staged experiment

you can use to declare experiment in code

__init__(model: torch.nn.modules.module.Module, loaders: OrderedDict[str, DataLoader], callbacks: Union[OrderedDict[str, Callback], List[Callback]] = None, logdir: str = None, stage: str = 'train', criterion: torch.nn.modules.module.Module = None, optimizer: torch.optim.optimizer.Optimizer = None, scheduler: torch.optim.lr_scheduler._LRScheduler = None, num_epochs: int = 1, valid_loader: str = 'valid', main_metric: str = 'loss', minimize_metric: bool = True, verbose: bool = False, check_run: bool = False, state_kwargs: Dict = None, checkpoint_data: Dict = None, distributed_params: Dict = None, monitoring_params: Dict = None, initial_seed: int = 42)

Parameters

• model (Model) – model

• loaders (dict) – dictionary containing one or several torch.utils.data.DataLoader for training and validation

• callbacks (List[catalyst.dl.Callback]) – list of callbacks

• logdir (str) – path to output directory

• stage (str) – current stage

• criterion (Criterion) – criterion function

• optimizer (Optimizer) – optimizer

• scheduler (Scheduler) – scheduler

• num_epochs (int) – number of experiment’s epochs

• valid_loader (str) – loader name used to calculate the metrics and save the checkpoints. For example, you can pass train and then the metrics will be taken from train loader.

• main_metric (str) – the key to the name of the metric by which the checkpoints will be selected.

• minimize_metric (bool) – flag to indicate whether the main_metric should be minimized.

• verbose (bool) – ff true, it displays the status of the training to the console.

• state_kwargs (dict) – additional state params to State

• checkpoint_data (dict) – additional data to save in checkpoint, for example: class_names, date_of_training, etc

• distributed_params (dict) – dictionary with the parameters for distributed and FP16 method

• monitoring_params (dict) – dict with the parameters for monitoring services

• initial_seed (int) – experiment’s initial seed value

property distributed_params

Dict with the parameters for distributed and FP16 method

get_callbacks(stage: str) → OrderedDict[str, Callback]

Returns the callbacks for a given stage

get_criterion(stage: str) → torch.nn.modules.module.Module

Returns the criterion for a given stage

get_loaders(stage: str, epoch: int = None) → OrderedDict[str, DataLoader]

Returns the loaders for a given stage

get_model(stage: str) → torch.nn.modules.module.Module

Returns the model for a given stage

get_optimizer(stage: str, model: torch.nn.modules.module.Module) → torch.optim.optimizer.Optimizer

Returns the optimizer for a given stage

get_scheduler(stage: str, optimizer=None) → torch.optim.lr_scheduler._LRScheduler

Returns the scheduler for a given stage

get_state_params(stage: str) → Mapping[str, Any]

Returns the state parameters for a given stage

property initial_seed

Experiment’s initial seed value

property logdir

Path to the directory where the experiment logs

property monitoring_params

Dict with the parameters for monitoring services

property stages

Experiment’s stage names (array with one value)

class catalyst.dl.experiment.config.ConfigExperiment(config: Dict)

Bases: catalyst.dl.core.experiment.Experiment

Experiment created from a configuration file

STAGE_KEYWORDS = ['criterion_params', 'optimizer_params', 'scheduler_params', 'data_params', 'transform_params', 'state_params', 'callbacks_params']

__init__(config: Dict)

Parameters

config (dict) – dictionary of parameters

property distributed_params

Dict with the parameters for distributed and FP16 methond

get_callbacks(stage: str) → OrderedDict[Callback]

Returns the callbacks for a given stage

get_criterion(stage: str) → torch.nn.modules.module.Module

Returns the criterion for a given stage

get_loaders(stage: str, epoch: int = None) → OrderedDict[str, DataLoader]

Returns the loaders for a given stage

get_model(stage: str)

Returns the model for a given stage

get_optimizer(stage: str, model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]]) → Union[torch.optim.optimizer.Optimizer, Dict[str, torch.optim.optimizer.Optimizer]]

Returns the optimizer for a given stage

Parameters

• stage (str) – stage name

• model (Union[Model, Dict[str, Model]]) – model or a dict of models

get_scheduler(stage: str, optimizer: torch.optim.optimizer.Optimizer) → torch.optim.lr_scheduler._LRScheduler

Returns the scheduler for a given stage

get_state_params(stage: str) → Mapping[str, Any]

Returns the state parameters for a given stage

get_transforms(stage: str = None, dataset: str = None) → Callable

Returns transform for a given stage & mode

Parameters

• stage (str) – stage name

• dataset (str) – dataset name (e.g. “train”, “valid”), will be used only if the value of _key_value` is True

property initial_seed

Experiment’s initial seed value

property logdir

Path to the directory where the experiment logs

property monitoring_params

Dict with the parameters for monitoring services

property stages

Experiment’s stage names

class catalyst.dl.experiment.supervised.SupervisedExperiment(model: torch.nn.modules.module.Module, loaders: OrderedDict[str, DataLoader], callbacks: Union[OrderedDict[str, Callback], List[Callback]] = None, logdir: str = None, stage: str = 'train', criterion: torch.nn.modules.module.Module = None, optimizer: torch.optim.optimizer.Optimizer = None, scheduler: torch.optim.lr_scheduler._LRScheduler = None, num_epochs: int = 1, valid_loader: str = 'valid', main_metric: str = 'loss', minimize_metric: bool = True, verbose: bool = False, check_run: bool = False, state_kwargs: Dict = None, checkpoint_data: Dict = None, distributed_params: Dict = None, monitoring_params: Dict = None, initial_seed: int = 42)

Bases: catalyst.dl.experiment.base.BaseExperiment

Supervised experiment

The main difference with BaseExperiment that it will add several callbacks by default if you haven’t.

Here are list of callbacks by default:

CriterionCallback:

measures loss with specified criterion.

OptimizerCallback:

abstraction over optimizer step.

SchedulerCallback:

only in case if you provided scheduler to your experiment does lr_scheduler.step

CheckpointCallback:

saves model and optimizer state each epoch callback to save/restore your model/criterion/optimizer/metrics.

ConsoleLogger:

standard Catalyst logger, translates state.*_metrics to console and text file

TensorboardLogger:

will write state.*_metrics to tensorboard

RaiseExceptionCallback:

will raise exception if needed

get_callbacks(stage: str) → OrderedDict[str, Callback]

Override of BaseExperiment.get_callbacks method. Will add several of callbacks by default in case they missed.

Parameters

stage (str) – name of stage. It should start with infer if you don’t need default callbacks, as they required only for training stages.

Returns

list of callbacks for experiment

Return type

List[Callback]

Runner

class catalyst.dl.runner.supervised.SupervisedRunner(model: torch.nn.modules.module.Module = None, device: Union[str, torch.device] = None, input_key: Any = 'features', output_key: Any = 'logits', input_target_key: str = 'targets')

Bases: catalyst.dl.core.runner.Runner

Runner for experiments with supervised model

__init__(model: torch.nn.modules.module.Module = None, device: Union[str, torch.device] = None, input_key: Any = 'features', output_key: Any = 'logits', input_target_key: str = 'targets')

Parameters

• model (Module) – Torch model object

• device (Device) – Torch device

• input_key (Any) – Key in batch dict mapping for model input

• output_key (Any) – Key in output dict model output will be stored under

• input_target_key (str) – Key in batch dict mapping for target

forward(batch, **kwargs)

Should not be called directly outside of runner. If your model has specific interface, override this method to use it

infer(model: torch.nn.modules.module.Module, loaders: OrderedDict[str, DataLoader], callbacks: Union[List[Callback], OrderedDict[str, Callback]] = None, verbose: bool = False, state_kwargs: Dict = None, fp16: Union[Dict, bool] = None, check: bool = False) → None

Makes the inference on the model.

Parameters

• model (Model) – model to infer

• loaders (dict) – dictionary containing one or several torch.utils.data.DataLoader for inference

• callbacks (List[catalyst.dl.Callback]) – list of inference callbacks

• verbose (bool) – ff true, it displays the status of the inference to the console.

• state_kwargs (dict) – additional state params to State

• fp16 (Union[Dict, bool]) – If not None, then sets inference to FP16. See https://nvidia.github.io/apex/amp.html#properties if fp16=True, params by default will be {"opt_level": "O1"}

• check (bool) – if True, then only checks that pipeline is working (3 epochs only)

predict_loader(model: torch.nn.modules.module.Module, loader: torch.utils.data.dataloader.DataLoader, resume: str = None, verbose: bool = False, state_kwargs: Dict = None, fp16: Union[Dict, bool] = None, check: bool = False) → Any

Makes a prediction on the whole loader with the specified model.

Parameters

• model (Model) – model to infer

• loader (DataLoader) – dictionary containing only one torch.utils.data.DataLoader for inference

• resume (str) – path to checkpoint for model

• verbose (bool) – ff true, it displays the status of the inference to the console.

• state_kwargs (dict) – additional state params to State

• fp16 (Union[Dict, bool]) – If not None, then sets inference to FP16. See https://nvidia.github.io/apex/amp.html#properties if fp16=True, params by default will be {"opt_level": "O1"}

• check (bool) – if True, then only checks that pipeline is working (3 epochs only)

trace(model: torch.nn.modules.module.Module = None, batch=None, logdir: str = None, loader: torch.utils.data.dataloader.DataLoader = None, method_name: str = 'forward', mode: str = 'eval', requires_grad: bool = False, fp16: Union[Dict, bool] = None, device: Union[str, torch.device] = 'cpu', predict_params: dict = None) → torch.jit.ScriptModule

Traces model using Torch Jit

Parameters

• model (Model) – model to trace

• batch – batch to forward through the model to trace

• logdir (str, optional) – If specified, the result will be written to the directory

• loader (DataLoader, optional) – if batch is not specified, the batch will be next(iter(loader))

• method_name (str) – model’s method name that will be traced

• mode (str) – train or eval

• requires_grad (bool) – flag to trace with gradients

• fp16 (Union[Dict, bool]) – If not None, then sets tracing params to FP16

• deivice (Device) – Torch deivice or a string

• predict_params (dict) – additional parameters for model forward

train(model: torch.nn.modules.module.Module, criterion: torch.nn.modules.module.Module, optimizer: torch.optim.optimizer.Optimizer, loaders: OrderedDict[str, DataLoader], logdir: str, callbacks: Union[List[Callback], OrderedDict[str, Callback]] = None, scheduler: torch.optim.lr_scheduler._LRScheduler = None, resume: str = None, num_epochs: int = 1, valid_loader: str = 'valid', main_metric: str = 'loss', minimize_metric: bool = True, verbose: bool = False, state_kwargs: Dict = None, checkpoint_data: Dict = None, fp16: Union[Dict, bool] = None, monitoring_params: Dict = None, check: bool = False) → None

Starts the training process of the model.

Parameters

• model (Model) – model to train

• criterion (Criterion) – criterion function for training

• optimizer (Optimizer) – optimizer for training

• loaders (dict) – dictionary containing one or several torch.utils.data.DataLoader for training and validation

• logdir (str) – path to output directory

• callbacks (List[catalyst.dl.Callback]) – list of callbacks

• scheduler (Scheduler) – scheduler for training

• resume (str) – path to checkpoint for model

• num_epochs (int) – number of training epochs

• valid_loader (str) – loader name used to calculate the metrics and save the checkpoints. For example, you can pass train and then the metrics will be taken from train loader.

• main_metric (str) – the key to the name of the metric by which the checkpoints will be selected.

• minimize_metric (bool) – flag to indicate whether the main_metric should be minimized.

• verbose (bool) – ff true, it displays the status of the training to the console.

• state_kwargs (dict) – additional state params to State

• checkpoint_data (dict) – additional data to save in checkpoint, for example: class_names, date_of_training, etc

• fp16 (Union[Dict, bool]) – If not None, then sets training to FP16. See https://nvidia.github.io/apex/amp.html#properties if fp16=True, params by default will be {"opt_level": "O1"}

• monitoring_params (dict) – If not None, then create monitoring through Alchemy or Weights&Biases. For example, {"token": "api_token", "experiment": "experiment_name"}

• check (bool) – if True, then only checks that pipeline is working (3 epochs only)

class catalyst.dl.runner.gan.MultiPhaseRunner(model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, device: Union[str, torch.device] = None, input_batch_keys: List[str] = None, registered_phases: Tuple[Tuple[str, Union[str, Callable]], ...] = None)

Bases: catalyst.dl.core.runner.Runner

Base Runner with multiple phases

__init__(model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, device: Union[str, torch.device] = None, input_batch_keys: List[str] = None, registered_phases: Tuple[Tuple[str, Union[str, Callable]], ...] = None)

Parameters

• model – gan models

• device – runner’s device

• input_batch_keys – list of strings of keys for batch elements, e.g. input_batch_keys = ["features", "targets"] and your DataLoader returns 2 tensors (images and targets) when state.input will be {"features": batch[0], "targets": batch[1]}

• registered_phases – Tuple of pairs (phase_name, phase_forward_function) phase_forward_function’s may be also str, in that case Runner should have method with same name, which will be called

forward(batch, **kwargs)

Forward call

class catalyst.dl.runner.gan.GanRunner(model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, device: Union[str, torch.device] = None, input_batch_keys: Optional[List[str]] = None, data_input_key: str = 'data', class_input_key: str = 'class_targets', noise_input_key: str = 'noise', fake_logits_output_key: str = 'fake_logits', real_logits_output_key: str = 'real_logits', fake_data_output_key: str = 'fake_data', fake_condition_keys: List[str] = None, real_condition_keys: List[str] = None, generator_train_phase: str = 'generator_train', discriminator_train_phase: str = 'discriminator_train', generator_model_key: str = 'generator', discriminator_model_key: str = 'discriminator')

Bases: catalyst.dl.runner.gan.MultiPhaseRunner

Runner with logic for single-generator single-discriminator GAN training

Various conditioning types, penalties and regularization (such as WGAN-GP) can be easily derived from this class

__init__(model: Union[torch.nn.modules.module.Module, Dict[str, torch.nn.modules.module.Module]] = None, device: Union[str, torch.device] = None, input_batch_keys: Optional[List[str]] = None, data_input_key: str = 'data', class_input_key: str = 'class_targets', noise_input_key: str = 'noise', fake_logits_output_key: str = 'fake_logits', real_logits_output_key: str = 'real_logits', fake_data_output_key: str = 'fake_data', fake_condition_keys: List[str] = None, real_condition_keys: List[str] = None, generator_train_phase: str = 'generator_train', discriminator_train_phase: str = 'discriminator_train', generator_model_key: str = 'generator', discriminator_model_key: str = 'discriminator')

Parameters

• model – Model

• device – Device

• input_batch_keys – list of strings of keys for batch elements, e.g. input_batch_keys = ["features", "targets"] and your DataLoader returns 2 tensors (images and targets) when state.input will be {"features": batch[0], "targets": batch[1]}

• data_input_key – real distribution to fit

• class_input_key – labels for real distribution

• noise_input_key – noise

• fake_logits_output_key – prediction scores of discriminator for fake data

• real_logits_output_key – prediction scores of discriminator for real data

• fake_data_output_key – generated data

• fake_condition_keys – list of all conditional inputs of discriminator (fake data conditions) (appear in same order as in generator model forward() call)

• real_condition_keys – list of all conditional inputs of discriminator (real data conditions) (appear in same order as in generator model forward() call)

• generator_train_phase (str) – name for generator training phase

• discriminator_train_phase (str) – name for discriminator training phase

• generator_model_key – name for generator model, e.g. “generator”

• discriminator_model_key – name for discriminator model, e.g. “discriminator”

Note

THIS RUNNER SUPPORTS ONLY EQUALLY CONDITIONED generator and discriminator (i.e. if generator is conditioned on 3 variables, discriminator must be conditioned on same 3 variables)

train(model: torch.nn.modules.module.Module, loaders: OrderedDict[str, DataLoader], callbacks: OrderedDict[str, Callback] = None, phase2callbacks: Dict[str, List[str]] = None, criterion: torch.nn.modules.module.Module = None, optimizer: torch.optim.optimizer.Optimizer = None, num_epochs: int = 1, main_metric: str = 'loss', minimize_metric: bool = True, state_kwargs: Dict = None, checkpoint_data: Dict = None, distributed_params: Dict = None, monitoring_params: Dict = None, logdir: str = None, verbose: bool = False, initial_seed: int = 42, check: bool = False) → None

Parameters

• model – models, usually generator and discriminator

• loaders – dictionary containing one or several torch.utils.data.DataLoader for training and validation

• callbacks – list of callbacks

• phase2callbacks – dictionary with lists of callback names which should be wrapped for appropriate phase for example: {“generator_train”: “loss_g”, “optim_g”} “loss_g” and “optim_g” callbacks from callbacks dict will be wrapped for “generator_train” phase in wrap_callbacks method

• criterion – criterion function

• optimizer – optimize

• num_epochs – number of experiment’s epochs the metrics and save the checkpoints.

• main_metric – the key to the name of the metric by which the checkpoints will be selected.

• minimize_metric – flag to indicate whether the main_metric should be minimized.

• state_kwargs – additional state params to RunnerState

• checkpoint_data – additional data to save in checkpoint, for example: class_names, date_of_training, etc

• distributed_params – dictionary with the parameters for distributed and FP16 method

• monitoring_params – dict with the parameters for monitoring services

• logdir – path to output directory

• verbose – if true, it displays the status of the training to the console.

• initial_seed – experiment’s initial seed value

• check – if True, then only checks that pipeline is working

• epochs only) ((3) –

Utils

catalyst.dl.utils.pipelines.clone_pipeline(template: str, out_dir: pathlib.Path) → None

Clones pipeline from empty pipeline template or from demo pipelines available in Git repos of Catalyst Team.

Parameters

• template (str) – type of pipeline you want to clone. empty/classification/segmentation

• out_dir (pathlib.Path) – path where pipeline directory should be cloned

Returns

None

catalyst.dl.utils.text.process_bert_output(bert_output, hidden_size: int, output_hidden_states: bool = False, pooling_groups: List[str] = None, mask: torch.Tensor = None, level: Union[int, str] = None)

Processed the output

catalyst.dl.utils.text.tokenize_text(text: str, tokenizer, max_length: int, strip: bool = True, lowercase: bool = True, remove_punctuation: bool = True) → Dict[str, numpy.array]

Tokenizes givin text

Parameters

• text (str) – text to tokenize

• tokenizer – Tokenizer instance from HuggingFace

• max_length (int) – maximum length of tokens

• strip (bool) – if true strips text before tokenizing

• lowercase (bool) – if true makes text lowercase before tokenizing

• remove_punctuation (bool) – if true removes string.punctuation from text before tokenizing

catalyst.dl.utils.torch.get_loader(data_source: Iterable[dict], open_fn: Callable, dict_transform: Callable = None, sampler=None, collate_fn: Callable = <function default_collate>, batch_size: int = 32, num_workers: int = 4, shuffle: bool = False, drop_last: bool = False)

Creates a DataLoader from given source and its open/transform params

Parameters

• data_source (Iterable[dict]) – and iterable containing your data annotations, (for example path to images, labels, bboxes, etc)

• open_fn (Callable) – function, that can open your annotations dict and transfer it to data, needed by your network (for example open image by path, or tokenize read string)

• dict_transform (callable) – transforms to use on dict (for example normalize image, add blur, crop/resize/etc)

• sampler (Sampler, optional) – defines the strategy to draw samples from the dataset

• collate_fn (callable, optional) – merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset

• batch_size (int, optional) – how many samples per batch to load

• num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process

• shuffle (bool, optional) – set to True to have the data reshuffled at every epoch (default: False).

• drop_last (bool, optional) – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default: False)

Returns

DataLoader with catalyst.data.ListDataset

catalyst.dl.utils.trace.trace_model(model: torch.nn.modules.module.Module, runner: catalyst.dl.core.runner.Runner, batch=None, method_name: str = 'forward', mode: str = 'eval', requires_grad: bool = False, opt_level: str = None, device: Union[str, torch.device] = 'cpu', predict_params: dict = None) → torch.jit.ScriptModule

Traces model using runner and batch

Parameters

• model – Model to trace

• runner – Model’s native runner that was used to train model

• batch – Batch to trace the model

• method_name (str) – Model’s method name that will be used as entrypoint during tracing

• mode (str) – Mode for model to trace (train or eval)

• requires_grad (bool) – Flag to use grads

• opt_level (str) – Apex FP16 init level, optional

• device (str) – Torch device

• predict_params (dict) – additional parameters for model forward

Returns

Traced model

Return type

(ScriptModule)

catalyst.dl.utils.trace.get_trace_name(method_name: str, mode: str = 'eval', requires_grad: bool = False, opt_level: str = None, additional_string: str = None)

Creates a file name for the traced model.

Parameters

• method_name (str) – model’s method name

• mode (str) – train or eval

• requires_grad (bool) – flag if model was traced with gradients

• opt_level (str) – opt_level if model was traced in FP16

• additional_string (str) – any additional information

catalyst.dl.utils.trace.load_traced_model(model_path: Union[str, pathlib.Path], device: Union[str, torch.device] = 'cpu', opt_level: str = None) → torch.jit.ScriptModule

Loads a traced model

Parameters

• model_path – Path to traced model

• device (str) – Torch device

• opt_level (str) – Apex FP16 init level, optional

Returns

Traced model

Return type

(ScriptModule)

catalyst.dl.utils.visualization.plot_metrics(logdir: Union[str, pathlib.Path], step: Optional[str] = 'epoch', metrics: Optional[List[str]] = None, height: Optional[int] = None, width: Optional[int] = None) → None

Plots your learning results.

Parameters

• logdir – the logdir that was specified during training.

• step – ‘batch’ or ‘epoch’ - what logs to show: for batches or for epochs

• metrics – list of metrics to plot. The loss should be specified as ‘loss’, learning rate = ‘_base/lr’ and other metrics should be specified as names in metrics dict that was specified during training

• height – the height of the whole resulting plot

• width – the width of the whole resulting plot

class catalyst.dl.utils.wizard.Wizard

Bases: object

Class for Catalyst Config API Wizard.

The instance of this class will be created and called from cli command: catalyst-dl init --interactive.

With help of this Wizard user will be able to setup pipeline from available templates and make choices of what predefined classes to use in different parts of pipeline.

__init__()

Initialization of instance of this class will print welcome message and logo of Catalyst in ASCII format. Also here we’ll save all classes of Catalyst own pipeline parts to be able to put user’s modules on top of lists to ease the choice.

run()

Walks user through predefined wizard steps

catalyst.dl.utils.wizard.run_wizard()

Method to initialize and run wizard

Registry

catalyst.dl.registry.Callback(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Criterion(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Optimizer(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Scheduler(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Module(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Model(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Sampler(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)

catalyst.dl.registry.Transform(factory: Union[Type, Callable[[...], Any]] = None, *factories: Union[Type, Callable[[...], Any]], name: str = None, **named_factories: Union[Type, Callable[[...], Any]]) → Union[Type, Callable[[...], Any]]

Adds factory to registry with it’s __name__ attribute or provided name. Signature is flexible.

Parameters

• factory – Factory instance

• factories – More instances

• name – Provided name for first instance. Use only when pass single instance.

• named_factories – Factory and their names as kwargs

Returns

First factory passed

Return type

(Factory)