Tools and Utilities

Tools

Frozen Class

Frozen class. Example of usage can be found in catalyst.core.runner.IRunner.

class catalyst.tools.frozen_class.FrozenClass

Bases: object

Class which prohibit __setattr__ on existing attributes.

Examples

>>> class IRunner(FrozenClass):

Time Manager

Simple timer.

class catalyst.tools.time_manager.TimeManager

Bases: object

@TODO: Docs. Contribution is welcome.

__init__()

@TODO: Docs. Contribution is welcome.

reset() → None

Reset all previous timers.

start(name: str) → None

Starts timer name.

Parameters

name (str) – name of a timer

stop(name: str) → None

Stops timer name.

Parameters

name (str) – name of a timer

Typing

All Catalyst custom types are defined in this module.

catalyst.tools.typing.Model

alias of torch.nn.modules.module.Module

catalyst.tools.typing.Criterion

alias of torch.nn.modules.module.Module

class catalyst.tools.typing.Optimizer(params, defaults)

Bases: object

Base class for all optimizers.

Warning

Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don’t satisfy those properties are sets and iterators over values of dictionaries.

Parameters

• params (iterable) – an iterable of torch.Tensor s or dict s. Specifies what Tensors should be optimized.

• defaults – (dict): a dict containing default values of optimization options (used when a parameter group doesn’t specify them).

add_param_group(param_group)

Add a param group to the Optimizer s param_groups.

This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.

Parameters

• param_group (dict) – Specifies what Tensors should be optimized along with group

• optimization options. (specific) –

load_state_dict(state_dict)

Loads the optimizer state.

Parameters

state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().

state_dict()

Returns the state of the optimizer as a dict.

It contains two entries:

• state - a dict holding current optimization state. Its content

  differs between optimizer classes.

• param_groups - a dict containing all parameter groups

step(closure)

Performs a single optimization step (parameter update).

Parameters

closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

Note

Unless otherwise specified, this function should not modify the .grad field of the parameters.

zero_grad()

Clears the gradients of all optimized torch.Tensor s.

catalyst.tools.typing.Scheduler

alias of torch.optim.lr_scheduler._LRScheduler

class catalyst.tools.typing.Dataset

Bases: object

An abstract class representing a Dataset.

All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite __getitem__(), supporting fetching a data sample for a given key. Subclasses could also optionally overwrite __len__(), which is expected to return the size of the dataset by many Sampler implementations and the default options of DataLoader.

Note

DataLoader by default constructs a index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided.

Meters

The meters from torchnet.meters.

Every meter implements catalyst.tools.meters.meter.Meter interface.

Meter

Meters provide a way to keep track of important statistics in an online manner.

class catalyst.tools.meters.meter.Meter

Bases: object

This class is abstract, but provides a standard interface for all meters to follow.

add(value)

Log a new value to the meter.

Parameters

value – Next result to include.

reset()

Resets the meter to default settings.

value()

Get the value of the meter in the current state.

AP Meter

The APMeter measures the average precision per class.

class catalyst.tools.meters.apmeter.APMeter

Bases: catalyst.tools.meters.meter.Meter

The APMeter is designed to operate on NxK Tensors output and target, and optionally a Nx1 Tensor weight where:

1. The output contains model output scores for N examples and K classes that ought to be higher when the model is more convinced that the example should be positively labeled, and smaller when the model believes the example should be negatively labeled (for instance, the output of a sigmoid function).

2. The target contains only values 0 (for negative examples) and 1 (for positive examples).

3. The weight ( > 0) represents weight for each sample.

__init__()

Constructor method for the APMeter class.

add(output: torch.Tensor, target: torch.Tensor, weight: torch.Tensor = None) → None

Add a new observation.

Parameters

• output (Tensor) – NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model. The probabilities should sum to one over all classes

• target (Tensor) – binary NxK tensort that encodes which of the K classes are associated with the N-th input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)

• weight (optional, Tensor) – Nx1 tensor representing the weight for each example (each weight > 0)

reset()

Resets the meter with empty member variables.

value() → torch.Tensor

Returns the model”s average precision for each class.

Returns

1xK tensor, with avg precision for each class k

Return type

torch.Tensor

AUC Meter

The AUCMeter measures the area under the receiver-operating characteristic (ROC) curve for binary classification problems. The area under the curve (AUC) can be interpreted as the probability that, given a randomly selected positive example and a randomly selected negative example, the positive example is assigned a higher score by the classification model than the negative example.

class catalyst.tools.meters.aucmeter.AUCMeter

Bases: catalyst.tools.meters.meter.Meter

The AUCMeter is designed to operate on one-dimensional Tensors output and target, where:

1. The output contains model output scores that ought to be higher when the model is more convinced that the example should be positively labeled, and smaller when the model believes the example should be negatively labeled (for instance, the output of a sigmoid function)

2. The target contains only values 0 (for negative examples) and 1 (for positive examples).

__init__()

Constructor method for the AUCMeter class.

add(output: torch.Tensor, target: torch.Tensor) → None

Update stored scores and targets.

Parameters

• output (Tensor) – one-dimensional tensor output

• target (Tensor) – one-dimensional tensor target

reset() → None

Reset stored scores and targets.

value()

Return metric values of AUC, TPR and FPR.

Returns

(AUC, TPR, FPR)

Return type

tuple of floats

Average Value Meter

Average value meter

class catalyst.tools.meters.averagevaluemeter.AverageValueMeter

Bases: catalyst.tools.meters.meter.Meter

Average value meter stores mean and standard deviation for population of input values. Meter updates are applied online, one value for each update. Values are not cached, only the last added.

__init__()

Constructor method for the AverageValueMeter class.

add(value, batch_size) → None

Add a new observation.

Updates of mean and std are going online, with Welford’s online algorithm.

Parameters

• value (float) – value for update, can be scalar number or PyTorch tensor

• batch_size (int) – batch size for update

Note

Because of algorithm design, you can update meter values with only one value a time.

reset()

Resets the meter to default settings.

value()

Returns meter values.

Returns

tuple of mean and std that have been updated online.

Return type

Tuple[float, float]

Class Error Meter

class catalyst.tools.meters.classerrormeter.ClassErrorMeter(topk=None, accuracy=False)

Bases: catalyst.tools.meters.meter.Meter

@TODO: Docs. Contribution is welcome.

__init__(topk=None, accuracy=False)

Constructor method for the AverageValueMeter class.

add(output, target) → None

@TODO: Docs. Contribution is welcome.

reset() → None

@TODO: Docs. Contribution is welcome.

value(k=-1)

@TODO: Docs. Contribution is welcome.

Confusion Meter

Maintains a confusion matrix for a given classification problem.

class catalyst.tools.meters.confusionmeter.ConfusionMeter(k: int, normalized: bool = False)

Bases: catalyst.tools.meters.meter.Meter

ConfusionMeter constructs a confusion matrix for a multi-class classification problems. It does not support multi-label, multi-class problems: for such problems, please use MultiLabelConfusionMeter.

__init__(k: int, normalized: bool = False)

Parameters

• k (int) – number of classes in the classification problem

• normalized (boolean) – determines whether or not the confusion matrix is normalized or not

add(predicted: torch.Tensor, target: torch.Tensor) → None

Computes the confusion matrix of K x K size where K is no of classes.

Parameters

• predicted (tensor) – Can be an N x K tensor of predicted scores obtained from the model for N examples and K classes or an N-tensor of integer values between 0 and K-1

• target (tensor) – Can be a N-tensor of integer values assumed to be integer values between 0 and K-1 or N x K tensor, where targets are assumed to be provided as one-hot vectors

reset() → None

Reset confusion matrix, filling it with zeros.

value()

Returns

Confusion matrix of K rows and K columns, where rows corresponds to ground-truth targets and columns corresponds to predicted targets.

Map Meter

The mAP meter measures the mean average precision over all classes.

class catalyst.tools.meters.mapmeter.mAPMeter

Bases: catalyst.tools.meters.meter.Meter

This meter is a wrapper for catalyst.tools.meters.apmeter.APMeter. The mAPMeter is designed to operate on NxK Tensors output and target, and optionally a Nx1 Tensor weight where:

1. The output contains model output scores for N examples and K classes that ought to be higher when the model is more convinced that the example should be positively labeled, and smaller when the model believes the example should be negatively labeled (for instance, the output of a sigmoid function)

2. The target contains only values 0 (for negative examples) and 1 (for positive examples)

3. The weight ( > 0) represents weight for each sample.

__init__()

Constructor method for the mAPMeter class.

add(output: torch.Tensor, target: torch.Tensor, weight: Optional[torch.Tensor] = None) → None

Update self.apmeter.

Parameters

• output (Tensor) – Model output scores as NxK tensor

• target (Tensor) – Target scores as NxK tensor

• weight (Tensor, optional) – Weight values for each sample as Nx1 Tensor

reset() → None

Reset self.apmeter.

value()

Returns mean of self.apmeter value.

Returns

mAP scalar tensor

Return type

torch.Tensor

Moving Average Value Meter

Moving average meter calculates average for moving window of values.

class catalyst.tools.meters.movingaveragevaluemeter.MovingAverageValueMeter(windowsize)

Bases: catalyst.tools.meters.meter.Meter

MovingAverageValueMeter stores mean and standard deviation for population of array that is handled like a queue during updates. Queue(window) is filled with zeros from the start by default. Meter updates are applied online, one value for each update. Meter values are moving average and moving standard deviation.

__init__(windowsize)

Parameters

windowsize (int) – size of window of values, which is continuous and ends on last updated element

add(value: float) → None

Adds observation sample.

Parameters

value (float) – scalar

reset() → None

Reset sum, number of elements, moving variance and zero out window.

value()

Return mean and standard deviation of window.

Returns

(window mean, window std)

Return type

tuple of floats

MSE Meter

MSE and RMSE meters.

class catalyst.tools.meters.msemeter.MSEMeter(root: bool = False)

Bases: catalyst.tools.meters.meter.Meter

This meter can handle MSE and RMSE. Root calculation can be toggled(not calculated by default).

__init__(root: bool = False)

Parameters

root (bool) – Toggle between calculation of RMSE (True) and MSE (False)

add(output: torch.Tensor, target: torch.Tensor) → None

Update squared error stored sum and number of elements.

Parameters

• output (Tensor) – Model output tensor or numpy array

• target (Tensor) – Target tensor or numpy array

reset() → None

Reset meter number of elements and squared error sum.

value() → float

Calculate MSE and return RMSE or MSE.

Returns

Root of MSE if self.root is True else MSE

Return type

float

Precision-Recall-F1 Meter

In this module precision, recall and F1 score calculations are defined in separate functions.

PrecisionRecallF1ScoreMeter can keep track for all three of these.

class catalyst.tools.meters.ppv_tpr_f1_meter.PrecisionRecallF1ScoreMeter(threshold=0.5)

Bases: catalyst.tools.meters.meter.Meter

Keeps track of global true positives, false positives, and false negatives for each epoch and calculates precision, recall, and F1-score based on those metrics. Currently, this meter works for binary cases only, please use multiple instances of this class for multi-label cases.

__init__(threshold=0.5)

Constructor method for the `` PrecisionRecallF1ScoreMeter`` class.

add(output: torch.Tensor, target: torch.Tensor) → None

Thresholds predictions and calculates the true positives, false positives, and false negatives in comparison to the target.

Parameters

• output (torch.Tensor) – prediction after activation function shape should be (batch_size, …), but works with any shape

• target (torch.Tensor) – label (binary), shape should be the same as output’s shape

reset() → None

Resets true positive, false positive and false negative counts to 0.

value()

Calculates precision/recall/f1 based on the current stored tp/fp/fn counts.

Returns

(precision, recall, f1)

Return type

tuple of floats

Utils

All utils are gathered in catalyst.utils for easier access.

Note

Everything from catalyst.contrib.utils is included in catalyst.utils

Checkpoint

catalyst.utils.checkpoint.pack_checkpoint(model=None, criterion=None, optimizer=None, scheduler=None, **kwargs)

@TODO: Docs. Contribution is welcome.

catalyst.utils.checkpoint.unpack_checkpoint(checkpoint, model=None, criterion=None, optimizer=None, scheduler=None) → None

Load checkpoint from file and unpack the content to a model (if not None), criterion (if not None), optimizer (if not None), scheduler (if not None).

Parameters

• checkpoint (str) – checkpoint to load

• model (torch.nn.Module) – model where should be updated state

• criterion – criterion where should be updated state

• optimizer – optimizer where should be updated state

• scheduler – scheduler where should be updated state

catalyst.utils.checkpoint.save_checkpoint(checkpoint: Dict, logdir: Union[pathlib.Path, str], suffix: str, is_best: bool = False, is_last: bool = False, special_suffix: str = '', saver_fn: Callable = <function save>)

Saving checkpoint to a file.

Parameters

• checkpoint (dict) – data to save.

• logdir (Path/str) – directory where checkpoint should be stored.

• suffix (str) – checkpoint file name.

• is_best (bool) – if True then also will be generated best checkpoint file.

• is_last (bool) – if True then also will be generated last checkpoint file.

• special_suffix (str) – suffix to use for saving best/last checkpoints.

• saver_fn (Callable) – function to use for saving data to file, default is torch.save

catalyst.utils.checkpoint.load_checkpoint(filepath: str)

Load checkpoint from path. :param filepath: checkpoint file to load :type filepath: str

Returns

checkpoint content

Components

catalyst.utils.components.process_components(model: torch.nn.modules.module.Module, criterion: torch.nn.modules.module.Module = None, optimizer: torch.optim.optimizer.Optimizer = None, scheduler: torch.optim.lr_scheduler._LRScheduler = None, distributed_params: Dict = None, device: Union[str, torch.device] = None) → Tuple[torch.nn.modules.module.Module, torch.nn.modules.module.Module, torch.optim.optimizer.Optimizer, torch.optim.lr_scheduler._LRScheduler, Union[str, torch.device]]

Returns the processed model, criterion, optimizer, scheduler and device.

Parameters

• model (Model) – torch model

• criterion (Criterion) – criterion function

• optimizer (Optimizer) – optimizer

• scheduler (Scheduler) – scheduler

• distributed_params (dict, optional) – dict with the parameters for distributed and FP16 method

• device (Device, optional) – device

Returns

tuple with processed model, criterion, optimizer, scheduler and device.

Raises

• ValueError – if device is None and TPU available, for using TPU need to manualy move model/optimizer/scheduler to a TPU device and pass device to a function.

• NotImplementedError – if model is not nn.Module or dict for multi-gpu, nn.ModuleDict for DataParallel not implemented yet

Config

catalyst.utils.config.load_config(path: Union[str, pathlib.Path], ordered: bool = False, data_format: str = None, encoding: str = 'utf-8') → Union[Dict, List]

Loads config by giving path. Supports YAML and JSON files.

Examples

>>> load(path="./config.yml", ordered=True)

Parameters

• path (str) – path to config file (YAML or JSON)

• ordered (bool) – if true the config will be loaded as OrderedDict

• data_format (str) – yaml, yml or json.

• encoding (str) – encoding to read the config

Returns

config

Return type

Union[Dict, List]

Raises

Exception – if path path doesn’t exists or file format is not YAML or JSON

Adapted from https://github.com/TezRomacH/safitty/blob/v1.2.0/safitty/parser.py#L63 which was adapted from https://github.com/catalyst-team/catalyst/blob/v19.03/catalyst/utils/config.py#L10

catalyst.utils.config.save_config(config: Union[Dict, List], path: Union[str, pathlib.Path], data_format: str = None, encoding: str = 'utf-8', ensure_ascii: bool = False, indent: int = 2) → None

Saves config to file. Path must be either YAML or JSON.

Parameters

• config (Union[Dict, List]) – config to save

• path (Union[str, Path]) – path to save

• data_format (str) – yaml, yml or json.

• encoding (str) – Encoding to write file. Default is utf-8

• ensure_ascii (bool) – Used for JSON, if True non-ASCII

• are escaped in JSON strings. (characters) –

• indent (int) – Used for JSON

Adapted from https://github.com/TezRomacH/safitty/blob/v1.2.0/safitty/parser.py#L110 which was adapted from https://github.com/catalyst-team/catalyst/blob/v19.03/catalyst/utils/config.py#L38

Distributed

catalyst.utils.distributed.check_ddp_wrapped(model: torch.nn.modules.module.Module) → bool

Checks whether model is wrapped with DataParallel/DistributedDataParallel.

catalyst.utils.distributed.check_apex_available() → bool

Checks if apex is available.

catalyst.utils.distributed.check_torch_distributed_initialized() → bool

Checks if torch.distributed is available and initialized.

catalyst.utils.distributed.check_slurm_available()

Checks if slurm is available.

catalyst.utils.distributed.assert_fp16_available() → None

Asserts for installed and available Apex FP16.

catalyst.utils.distributed.initialize_apex(model, optimizer=None, **distributed_params)

@TODO: Docs. Contribution is welcome.

catalyst.utils.distributed.get_nn_from_ddp_module(model: torch.nn.modules.module.Module) → torch.nn.modules.module.Module

Return a real model from a torch.nn.DataParallel, torch.nn.parallel.DistributedDataParallel, or apex.parallel.DistributedDataParallel.

Parameters

model – A model, or DataParallel wrapper.

Returns

A model

catalyst.utils.distributed.get_rank() → int

Returns the rank of the current worker.

Returns

rank if torch.distributed is initialized, otherwise -1

Return type

int

catalyst.utils.distributed.get_distributed_mean(value: Union[float, torch.Tensor])

Computes distributed mean among all nodes.

catalyst.utils.distributed.get_distributed_env(local_rank, rank, world_size, use_cuda_visible_devices=True)

@TODO: Docs. Contribution is welcome.

catalyst.utils.distributed.get_distributed_params()

@TODO: Docs. Contribution is welcome.

catalyst.utils.distributed.get_slurm_params()

@TODO: Docs. Contribution is welcome.

catalyst.utils.distributed.is_wrapped_with_ddp(model: torch.nn.modules.module.Module) → bool

Checks whether model is wrapped with DataParallel/DistributedDataParallel.

Deprecated since version 20.05: This will be removed in 20.06. Use check_ddp_wrapped instead.

catalyst.utils.distributed.is_torch_distributed_initialized() → bool

Checks if torch.distributed is available and initialized.

Deprecated since version 20.05: This will be removed in 20.06. Use check_torch_distributed_initialized instead.

catalyst.utils.distributed.is_slurm_available() → bool

Checks if slurm is available.

Deprecated since version 20.05: This will be removed in 20.06. Use check_slurm_available instead.

catalyst.utils.distributed.is_apex_available() → bool

Checks if apex is available.

Deprecated since version 20.05: This will be removed in 20.06. Use check_apex_available instead.

Hash

catalyst.utils.hash.get_hash(obj: Any) → str

Creates unique hash from object following way: - Represent obj as sting recursively - Hash this string with sha256 hash function - encode hash with url-safe base64 encoding

Parameters

obj – object to hash

Returns

base64-encoded string

catalyst.utils.hash.get_short_hash(o) → str

@TODO: Docs. Contribution is welcome.

Initialization

catalyst.utils.initialization.get_optimal_inner_init(nonlinearity: torch.nn.modules.module.Module, **kwargs) → Callable[[torch.nn.modules.module.Module], None]

Create initializer for inner layers based on their activation function (nonlinearity).

Parameters

• nonlinearity – non-linear activation

• **kwargs – extra kwargs

Returns

optimal initialization function

Raises

NotImplementedError – if nonlinearity is out of sigmoid, tanh, relu, `leaky_relu

catalyst.utils.initialization.outer_init(layer: torch.nn.modules.module.Module) → None

Initialization for output layers of policy and value networks typically used in deep reinforcement learning literature.

Parameters

layer – torch nn.Module instance

catalyst.utils.initialization.reset_weights_if_possible(module: torch.nn.modules.module.Module)

Resets module parameters if possible.

Parameters

module – Module to reset.

Misc

catalyst.utils.misc.copy_directory(input_dir: pathlib.Path, output_dir: pathlib.Path) → None

Recursively copies the input directory.

Parameters

• input_dir (Path) – input directory

• output_dir (Path) – output directory

catalyst.utils.misc.format_metric(name: str, value: float) → str

Format metric.

Metric will be returned in the scientific format if 4 decimal chars are not enough (metric value lower than 1e-4).

Parameters

• name (str) – metric name

• value (float) – value of metric

Returns

formatted metric

Return type

str

catalyst.utils.misc.get_fn_default_params(fn: Callable[[...], Any], exclude: List[str] = None)

Return default parameters of Callable.

Parameters

• fn (Callable[.., Any]) – target Callable

• exclude (List[str]) – exclude list of parameters

Returns

contains default parameters of fn

Return type

dict

catalyst.utils.misc.get_fn_argsnames(fn: Callable[[...], Any], exclude: List[str] = None)

Return parameter names of Callable.

Parameters

• fn (Callable[.., Any]) – target Callable

• exclude (List[str]) – exclude list of parameters

Returns

contains parameter names of fn

Return type

list

catalyst.utils.misc.get_utcnow_time(format: str = None) → str

Return string with current utc time in chosen format.

Parameters

format (str) – format string. if None “%y%m%d.%H%M%S” will be used.

Returns

formatted utc time string

Return type

str

catalyst.utils.misc.is_exception(ex: Any) → bool

Check if the argument is of Exception type.

catalyst.utils.misc.maybe_recursive_call(object_or_dict, method: str, recursive_args=None, recursive_kwargs=None, **kwargs)

Calls the method recursively for the object_or_dict.

Parameters

• object_or_dict (Any) – some object or a dictionary of objects

• method (str) – method name to call

• recursive_args – list of arguments to pass to the method

• recursive_kwargs – list of key-arguments to pass to the method

• **kwargs – Arbitrary keyword arguments

Returns

result of method call

catalyst.utils.misc.fn_ends_with_pass(fn: Callable[[...], Any])

Check that function end with pass statement (probably does nothing in any way). Mainly used to filter callbacks with empty on_{event} methods.

Parameters

fn (Callable[.., Any]) – target Callable

Returns

True if there is pass in the first indentation level of fn and nothing happens before it, False in any other case.

Return type

bool

Numpy

catalyst.utils.numpy.get_one_hot(label: int, num_classes: int, smoothing: float = None) → numpy.ndarray

Applies OneHot vectorization to a giving scalar, optional with label smoothing as described in Bag of Tricks for Image Classification with Convolutional Neural Networks.

Parameters

• label (int) – scalar value to be vectorized

• num_classes (int) – total number of classes

• smoothing (float, optional) – if specified applies label smoothing from Bag of Tricks for Image Classification with Convolutional Neural Networks paper

Returns

a one-hot vector with shape (num_classes,)

Return type

np.ndarray

Parser

catalyst.utils.parser.parse_config_args(*, config, args, unknown_args)

@TODO: Docs. Contribution is welcome.

catalyst.utils.parser.parse_args_uargs(args, unknown_args)

Function for parsing configuration files.

Parameters

• args – recognized arguments

• unknown_args – unrecognized arguments

Returns

updated arguments, dict with config

Return type

tuple

Scripts

catalyst.utils.scripts.import_module(expdir: pathlib.Path)

@TODO: Docs. Contribution is welcome

catalyst.utils.scripts.dump_code(expdir, logdir)

@TODO: Docs. Contribution is welcome

catalyst.utils.scripts.dump_python_files(src, dst)

@TODO: Docs. Contribution is welcome

catalyst.utils.scripts.import_experiment_and_runner(expdir: pathlib.Path)

@TODO: Docs. Contribution is welcome

catalyst.utils.scripts.dump_base_experiment_code(src: pathlib.Path, dst: pathlib.Path)

@TODO: Docs. Contribution is welcome

catalyst.utils.scripts.distributed_cmd_run(worker_fn: Callable, distributed: bool = True, *args, **kwargs) → None

Distributed run

Parameters

• worker_fn (Callable) – worker fn to run in distributed mode

• distributed (bool) – distributed flag

• args – additional parameters for worker_fn

• kwargs – additional key-value parameters for worker_fn

Seed

catalyst.utils.seed.set_global_seed(seed: int) → None

Sets random seed into PyTorch, TensorFlow, Numpy and Random.

Parameters

seed – random seed

Sys

catalyst.utils.sys.get_environment_vars() → Dict[str, Any]

Creates a dictionary with environment variables.

Returns

environment variables

Return type

Dict

catalyst.utils.sys.list_conda_packages() → str

Lists conda installed packages.

Returns

list with conda installed packages

Return type

str

catalyst.utils.sys.list_pip_packages() → str

Lists pip installed packages.

Returns

string with pip installed packages

Return type

str

catalyst.utils.sys.dump_environment(experiment_config: Dict, logdir: str, configs_path: List[str] = None) → None

Saves config, environment variables and package list in JSON into logdir.

Parameters

• experiment_config (dict) – experiment config

• logdir (str) – path to logdir

• configs_path – path(s) to config

Torch

catalyst.utils.torch.get_optimizable_params(model_or_params)

Returns all the parameters that requires gradients.

catalyst.utils.torch.get_optimizer_momentum(optimizer: torch.optim.optimizer.Optimizer) → float

Get momentum of current optimizer.

Parameters

optimizer – PyTorch optimizer

Returns

momentum at first param group

Return type

float

catalyst.utils.torch.set_optimizer_momentum(optimizer: torch.optim.optimizer.Optimizer, value: float, index: int = 0)

Set momentum of index ‘th param group of optimizer to value.

Parameters

• optimizer – PyTorch optimizer

• value (float) – new value of momentum

• index (int, optional) – integer index of optimizer’s param groups, default is 0

catalyst.utils.torch.get_device() → torch.device

Simple returning the best available device (TPU > GPU > CPU).

catalyst.utils.torch.get_available_gpus()

Array of available GPU ids.

Examples

>>> os.environ["CUDA_VISIBLE_DEVICES"] = "0,2"
>>> get_available_gpus()
[0, 2]

>>> os.environ["CUDA_VISIBLE_DEVICES"] = "0,-1,1"
>>> get_available_gpus()
[0]

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
>>> get_available_gpus()
[]

>>> os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
>>> get_available_gpus()
[]

Returns

available GPU ids

Return type

iterable

catalyst.utils.torch.get_activation_fn(activation: str = None)

Returns the activation function from torch.nn by its name.

catalyst.utils.torch.any2device(value, device: Union[str, torch.device])

Move tensor, list of tensors, list of list of tensors, dict of tensors, tuple of tensors to target device.

Parameters

• value – Object to be moved

• device (Device) – target device ids

Returns

Same structure as value, but all tensors and np.arrays moved to device

catalyst.utils.torch.prepare_cudnn(deterministic: bool = None, benchmark: bool = None) → None

Prepares CuDNN benchmark and sets CuDNN to be deterministic/non-deterministic mode

Parameters

• deterministic (bool) – deterministic mode if running in CuDNN backend.

• benchmark (bool) – If True use CuDNN heuristics to figure out which algorithm will be most performant for your model architecture and input. Setting it to False may slow down your training.

catalyst.utils.torch.process_model_params(model: torch.nn.modules.module.Module, layerwise_params: Dict[str, dict] = None, no_bias_weight_decay: bool = True, lr_scaling: float = 1.0) → List[Union[torch.nn.parameter.Parameter, dict]]

Gains model parameters for torch.optim.Optimizer.

Parameters

• model (torch.nn.Module) – Model to process

• layerwise_params (Dict) – Order-sensitive dict where each key is regex pattern and values are layer-wise options for layers matching with a pattern

• no_bias_weight_decay (bool) – If true, removes weight_decay for all bias parameters in the model

• lr_scaling (float) – layer-wise learning rate scaling, if 1.0, learning rates will not be scaled

Returns

parameters for an optimizer

Return type

iterable

Example:

>>> model = catalyst.contrib.models.segmentation.ResnetUnet()
>>> layerwise_params = collections.OrderedDict([
>>>     ("conv1.*", dict(lr=0.001, weight_decay=0.0003)),
>>>     ("conv.*", dict(lr=0.002))
>>> ])
>>> params = process_model_params(model, layerwise_params)
>>> optimizer = torch.optim.Adam(params, lr=0.0003)

catalyst.utils.torch.get_requires_grad(model: torch.nn.modules.module.Module)

Gets the requires_grad value for all model parameters.

Example:

>>> model = SimpleModel()
>>> requires_grad = get_requires_grad(model)

Parameters

model (torch.nn.Module) – model

Returns

value

Return type

requires_grad (Dict[str, bool])

catalyst.utils.torch.set_requires_grad(model: torch.nn.modules.module.Module, requires_grad: Union[bool, Dict[str, bool]])

Sets the requires_grad value for all model parameters.

Example:

>>> model = SimpleModel()
>>> set_requires_grad(model, requires_grad=True)
>>> # or
>>> model = SimpleModel()
>>> set_requires_grad(model, requires_grad={""})

Parameters

• model (torch.nn.Module) – model

• requires_grad (Union[bool, Dict[str, bool]]) – value

catalyst.utils.torch.get_network_output(net: torch.nn.modules.module.Module, *input_shapes_args, **input_shapes_kwargs)

# noqa: D202 For each input shape returns an output tensor

Examples

>>> net = nn.Linear(10, 5)
>>> utils.get_network_output(net, (1, 10))
tensor([[[-0.2665,  0.5792,  0.9757, -0.5782,  0.1530]]])

Parameters

• net (Model) – the model

• *input_shapes_args – variable length argument list of shapes

• **input_shapes_kwargs – key-value arguemnts of shapes

Returns

tensor with network output

catalyst.utils.torch.detach(tensor: torch.Tensor) → numpy.ndarray

Detach a pytorch tensor from graph and convert it to numpy array

Parameters

tensor – PyTorch tensor

Returns

numpy ndarray

catalyst.utils.torch.trim_tensors(tensors)

Trim padding off of a batch of tensors to the smallest possible length. Should be used with catalyst.data.DynamicLenBatchSampler.

Adapted from Dynamic minibatch trimming to improve BERT training speed.

Parameters

tensors ([torch.tensor]) – list of tensors to trim.

Returns

list of trimmed tensors.

Return type

List[torch.tensor]

catalyst.utils.torch.normalize(samples: torch.Tensor) → torch.Tensor

Parameters

samples – tensor with shape of [n_samples, features_dim]

Returns

normalized tensor with the same shape

Metrics

Accuracy

Various accuracy metrics:

• accuracy()

• multi_label_accuracy()

catalyst.utils.metrics.accuracy.accuracy(outputs: torch.Tensor, targets: torch.Tensor, topk: Sequence[int] = (1, ), activation: Optional[str] = None) → Sequence[torch.Tensor]

Computes multi-class accuracy@topk for the specified values of topk.

Parameters

• outputs (torch.Tensor) – model outputs, logits with shape [bs; num_classes]

• targets (torch.Tensor) – ground truth, labels with shape [bs; 1]

• activation (str) – activation to use for model output

• topk (Sequence[int]) – topk for accuracy@topk computing

Returns

list with computed accuracy@topk

catalyst.utils.metrics.accuracy.multi_label_accuracy(outputs: torch.Tensor, targets: torch.Tensor, threshold: Union[float, torch.Tensor], activation: Optional[str] = None) → torch.Tensor

Computes multi-label accuracy for the specified activation and threshold.

Parameters

• outputs (torch.Tensor) – NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.

• targets (torch.Tensor) – binary NxK tensort that encodes which of the K classes are associated with the N-th input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)

• threshold (float) – threshold for for model output

• activation (str) – activation to use for model output

Returns

computed multi-label accuracy

Computes multi-label accuracy for the specified activation and threshold.

param outputs

NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.

type outputs

torch.Tensor

param targets

binary NxK tensort that encodes which of the K classes are associated with the N-th input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)

type targets

torch.Tensor

param threshold

threshold for for model output

type threshold

float

param activation

activation to use for model output

type activation

str

returns

computed multi-label accuracy

AUC

catalyst.utils.metrics.auc.auc(outputs: torch.Tensor, targets: torch.Tensor) → torch.Tensor

AUC metric.

Parameters

• outputs – [bs; num_classes] estimated scores from a model.

• targets – [bs; num_classes] ground truth (correct) target values.

Returns

Tensor with [num_classes] shape of per-class-aucs

Return type

torch.Tensor

CMC score

Function to count CMC from distance matrix and conformity matrix.

param distances

distance matrix shape of (n_embeddings_x, n_embeddings_y)

param conformity_matrix

binary matrix with 1 on same label pos and 0 otherwise

param topk

number of top examples for cumulative score counting

returns

cmc score

catalyst.utils.metrics.cmc_score.cmc_score(query_embeddings: torch.Tensor, gallery_embeddings: torch.Tensor, conformity_matrix: torch.Tensor, topk: int = 1) → float

Function to count CMC score from query and gallery embeddings.

Parameters

• query_embeddings – tensor shape of (n_embeddings, embedding_dim) embeddings of the objects in querry

• gallery_embeddings – tensor shape of (n_embeddings, embedding_dim) embeddings of the objects in gallery

• conformity_matrix – binary matrix with 1 on same label pos and 0 otherwise

• topk – number of top examples for cumulative score counting

Returns

cmc score

catalyst.utils.metrics.cmc_score.cmc_score_count(distances: torch.Tensor, conformity_matrix: torch.Tensor, topk: int = 1) → float

Function to count CMC from distance matrix and conformity matrix.

Parameters

• distances – distance matrix shape of (n_embeddings_x, n_embeddings_y)

• conformity_matrix – binary matrix with 1 on same label pos and 0 otherwise

• topk – number of top examples for cumulative score counting

Returns

cmc score

Dice

Dice metric.

catalyst.utils.metrics.dice.dice(outputs: torch.Tensor, targets: torch.Tensor, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Computes the dice metric.

Parameters

• outputs (list) – a list of predicted elements

• targets (list) – a list of elements that are to be predicted

• eps (float) – epsilon

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]

Returns

Dice score

Return type

float

catalyst.utils.metrics.dice.calculate_dice(true_positives: numpy.array, false_positives: numpy.array, false_negatives: numpy.array) → numpy.array

Calculate list of Dice coefficients.

Parameters

• true_positives – true positives numpy tensor

• false_positives – false positives numpy tensor

• false_negatives – false negatives numpy tensor

Returns

dice score

Return type

np.array

Raises

ValueError – if dice is out of [0; 1] bounds

F1 score

F1 score.

catalyst.utils.metrics.f1_score.f1_score(outputs: torch.Tensor, targets: torch.Tensor, beta: float = 1.0, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• outputs (torch.Tensor) – A list of predicted elements

• targets (torch.Tensor) – A list of elements that are to be predicted

• eps (float) – epsilon to avoid zero division

• beta (float) – beta param for f_score

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]

Returns

F_1 score

Return type

float

Focal

Focal losses:

• sigmoid_focal_loss()

• reduced_focal_loss()

catalyst.utils.metrics.focal.sigmoid_focal_loss(outputs: torch.Tensor, targets: torch.Tensor, gamma: float = 2.0, alpha: float = 0.25, reduction: str = 'mean')

Compute binary focal loss between target and output logits.

Parameters

• outputs – tensor of arbitrary shape

• targets – tensor of the same shape as input

• gamma – gamma for focal loss

• alpha – alpha for focal loss

• reduction (string, optional) – specifies the reduction to apply to the output: "none" | "mean" | "sum" | "batchwise_mean". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed.

Returns

computed loss

Source: https://github.com/BloodAxe/pytorch-toolbelt

catalyst.utils.metrics.focal.reduced_focal_loss(outputs: torch.Tensor, targets: torch.Tensor, threshold: float = 0.5, gamma: float = 2.0, reduction='mean') → torch.Tensor

Compute reduced focal loss between target and output logits.

It has been proposed in Reduced Focal Loss: 1st Place Solution to xView object detection in Satellite Imagery paper.

Note

size_average and reduce params are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction.

Source: https://github.com/BloodAxe/pytorch-toolbelt

Parameters

• outputs – tensor of arbitrary shape

• targets – tensor of the same shape as input

• threshold – threshold for focal reduction

• gamma – gamma for focal reduction

• reduction (string, optional) – specifies the reduction to apply to the output: "none" | "mean" | "sum" | "batchwise_mean". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. "batchwise_mean" computes mean loss per sample in batch. Default: “mean”

Returns: # noqa: DAR201

torch.Tensor: computed loss

IoU

IoU metric. Jaccard metric refers to IoU here, same functionality.

catalyst.utils.metrics.iou.iou(outputs: torch.Tensor, targets: torch.Tensor, classes: List[str] = None, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid') → torch.Tensor

Parameters

• outputs (torch.Tensor) – A list of predicted elements

• targets (torch.Tensor) – A list of elements that are to be predicted

• classes (List[str]) – if classes are specified we reduce across all dims except channels

• eps (float) – epsilon to avoid zero division

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]

Returns

IoU (Jaccard) score(s)

Return type

Union[float, List[float]]

catalyst.utils.metrics.iou.jaccard(outputs: torch.Tensor, targets: torch.Tensor, classes: List[str] = None, eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid') → torch.Tensor

Parameters

• outputs (torch.Tensor) – A list of predicted elements

• targets (torch.Tensor) – A list of elements that are to be predicted

• classes (List[str]) – if classes are specified we reduce across all dims except channels

• eps (float) – epsilon to avoid zero division

• threshold (float) – threshold for outputs binarization

• activation (str) – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]

Returns

IoU (Jaccard) score(s)

Return type

Union[float, List[float]]

Precision

Computes the average precision.

param outputs

NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.

type outputs

torch.Tensor

param targets

binary NxK tensort that encodes which of the K classes are associated with the N-th input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)

type targets

torch.Tensor

param weights

importance for each sample

type weights

torch.Tensor

returns

tensor of [K; ] shape, with average precision for K classes

rtype

torch.Tensor

Functional

catalyst.utils.metrics.functional.get_default_topk_args(num_classes: int) → Sequence[int]

Calculate list params for Accuracy@k and mAP@k.

Examples

>>> get_default_topk_args(num_classes=4)
>>> [1, 3]
>>> get_default_topk_args(num_classes=8)
>>> [1, 3, 5]

Parameters

num_classes (int) – number of classes

Returns

array of accuracy arguments

Return type

iterable

catalyst.utils.metrics.functional.preprocess_multi_label_metrics(outputs: torch.Tensor, targets: torch.Tensor, weights: Optional[torch.Tensor] = None) → Tuple[torch.Tensor, torch.Tensor, torch.Tensor]

General preprocessing and check for multi-label-based metrics.

Parameters

• outputs (torch.Tensor) – NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.

• targets (torch.Tensor) – binary NxK tensor that encodes which of the K classes are associated with the N-th input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)

• weights (torch.Tensor) – importance for each sample

Returns

processed outputs and targets with [batch_size; num_classes] shape

catalyst.utils.metrics.functional.wrap_class_metric2dict(metric_fn: Callable, class_args: Sequence[str] = None) → Callable

# noqa: D202 Logging wrapper for metrics with torch.Tensor output and [num_classes] shape. Computes the metric and sync each element from the output Tensor with passed class argument.

Parameters

• metric_fn (Callable) – metric function to compute

• class_args (Sequence[str]) – class names for logging. default: None - class indexes will be used.

Returns

wrapped metric function with List[Dict] output

Return type

(Callable)

catalyst.utils.metrics.functional.wrap_topk_metric2dict(metric_fn: Callable, topk_args: Sequence[int]) → Callable

Logging wrapper for metrics with Sequence[Union[torch.Tensor, int, float, Dict]] output. Computes the metric and sync each element from the output sequence with passed topk argument.

Parameters

• metric_fn (Callable) – metric function to compute

• topk_args (Sequence[int]) – topk args to sync outputs with

Returns

wrapped metric function with List[Dict] output

Return type

(Callable)

Raises

NotImplementedError – if metrics returned values are out of torch.Tensor, int, float, Dict union.