Contrib

Datasets

MNIST

class catalyst.contrib.datasets.mnist.MNIST(root, train=True, transform=None, target_transform=None, download=False)

Bases: torch.utils.data.dataset.Dataset

MNIST Dataset.

__init__(root, train=True, transform=None, target_transform=None, download=False)

Parameters

• root – Root directory of dataset where MNIST/processed/training.pt and MNIST/processed/test.pt exist.

• train (bool, optional) – If True, creates dataset from training.pt, otherwise from test.pt.

• download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.

• transform (callable, optional) – A function/transform that takes in an image and returns a transformed version.

• target_transform (callable, optional) – A function/transform that takes in the target and transforms it.

property class_to_idx

Docs. Contribution is welcome.

Type

@TODO

classes = ['0 - zero', '1 - one', '2 - two', '3 - three', '4 - four', '5 - five', '6 - six', '7 - seven', '8 - eight', '9 - nine']

download()

Download the MNIST data if it doesn’t exist in processed_folder.

extra_repr()

@TODO: Docs. Contribution is welcome.

property processed_folder

Docs. Contribution is welcome.

Type

@TODO

property raw_folder

Docs. Contribution is welcome.

Type

@TODO

resources = [('http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz', 'f68b3c2dcbeaaa9fbdd348bbdeb94873'), ('http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz', 'd53e105ee54ea40749a09fcbcd1e9432'), ('http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz', '9fb629c4189551a2d022fa330f9573f3'), ('http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz', 'ec29112dd5afa0611ce80d1b7f02629c')]

test_file = 'test.pt'

training_file = 'training.pt'

class catalyst.contrib.datasets.mnist.MnistMLDataset(**kwargs)

Bases: catalyst.data.dataset.metric_learning.MetricLearningTrainDataset, catalyst.contrib.datasets.mnist.MNIST

Simple wrapper for MNIST dataset for metric learning train stage. This dataset can be used only for training. For test stage use MnistQGDataset.

For this dataset we use only training part of the MNIST and only those images that are labeled as 0, 1, 2, 3, 4.

__init__(**kwargs)

Raises

ValueError – if train argument is False (MnistMLDataset should be used only for training)

classes = ['0 - zero', '1 - one', '2 - two', '3 - three', '4 - four']

get_labels() → List[int]

Returns

labels of digits

class catalyst.contrib.datasets.mnist.MnistQGDataset(root: str, transform: Optional[Callable] = None, gallery_fraq: Optional[float] = 0.2)

Bases: catalyst.data.dataset.metric_learning.QueryGalleryDataset

MNIST for metric learning with query and gallery split. MnistQGDataset should be used for test stage.

For this dataset we used only test part of the MNIST and only those images that are labeled as 5, 6, 7, 8, 9.

__init__(root: str, transform: Optional[Callable] = None, gallery_fraq: Optional[float] = 0.2) → None

Parameters

• root – root directory for storing dataset

• transform – transform

• gallery_fraq – gallery size

classes = ['5 - five', '6 - six', '7 - seven', '8 - eight', '9 - nine']

property data

Images from MNIST

property gallery_size

Query Gallery dataset should have gallery_size property

property query_size

Query Gallery dataset should have query_size property

property targets

Labels of digits

MovieLens

class catalyst.contrib.datasets.movielens.MovieLens(root, train=True, download=False, min_rating=0.0)

Bases: torch.utils.data.dataset.Dataset

MovieLens data sets were collected by the GroupLens Research Project at the University of Minnesota.

This data set consists of: * 100,000 ratings (1-5) from 943 users on 1682 movies. * Each user has rated at least 20 movies. * Simple demographic info for the users (age, gender, occupation, zip)

The data was collected through the MovieLens web site (movielens.umn.edu) during the seven-month period from September 19th, 1997 through April 22nd, 1998. This data has been cleaned up - users who had less than 20 ratings or did not have complete demographic information were removed from this data set. Detailed descriptions of the data file can be found at the end of this file.

Neither the University of Minnesota nor any of the researchers involved can guarantee the correctness of the data, its suitability for any particular purpose, or the validity of results based on the use of the data set. The data set may be used for any research purposes under the following conditions: * The user may not state or imply any endorsement from the University of Minnesota or the GroupLens Research Group. * The user must acknowledge the use of the data set in publications resulting from the use of the data set (see below for citation information). * The user may not redistribute the data without separate permission. * The user may not use this information for any commercial or revenue-bearing purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota.

If you have any further questions or comments, please contact GroupLens <grouplens-info@cs.umn.edu>. http://files.grouplens.org/datasets/movielens/ml-100k-README.txt

__init__(root, train=True, download=False, min_rating=0.0)

Parameters

• root (string) – Root directory of dataset where MovieLens/processed/training.pt and MovieLens/processed/test.pt exist.

• train (bool, optional) – If True, creates dataset from training.pt, otherwise from test.pt.

• download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.

• min_rating (float, optional) – Minimum rating to include in the interaction matrix

filename = 'ml-100k.zip'

property processed_folder

Create the folder for the processed files

property raw_folder

Create raw folder for data download

resources = ('http://files.grouplens.org/datasets/movielens/ml-100k.zip', '0e33842e24a9c977be4e0107933c0723')

test_file = 'test.pt'

training_file = 'training.pt'

Computer Vision

ImageClassificationDataset

class catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.data.cv.dataset.ImageFolderDataset

Base class for datasets with the following structure:

path/to/dataset/
|-- train/
|   |-- class1/  # folder of N images
|   |   |-- train_image11
|   |   |-- train_image12
|   |   ...
|   |   `-- train_image1N
|   ...
|   `-- classM/  # folder of K images
|       |-- train_imageM1
|       |-- train_imageM2
|       ...
|       `-- train_imageMK
`-- val/
    |-- class1/  # folder of P images
    |   |-- val_image11
    |   |-- val_image12
    |   ...
    |   `-- val_image1P
    ...
    `-- classM/  # folder of T images
        |-- val_imageT1
        |-- val_imageT2
        ...
        `-- val_imageMT

__init__(root: str, train: bool = True, download: bool = False, **kwargs)

Constructor method for the ImageClassificationDataset class.

Parameters

• root – root directory of dataset

• train – if True, creates dataset from train/ subfolder, otherwise from val/

• download – if True, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again

• **kwargs –

resources = None

Imagenette

class catalyst.contrib.datasets.cv.imagenette.Imagenette(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagenette Dataset.

name = 'imagenette2'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz', '43b0d8047b7501984c47ae3c08110b62')]

class catalyst.contrib.datasets.cv.imagenette.Imagenette160(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagenette Dataset with images resized so that the shortest size is 160 px.

name = 'imagenette2-160'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagenette2-160.tgz', '0edfc972b5c9817ac36517c0057f3869')]

class catalyst.contrib.datasets.cv.imagenette.Imagenette320(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagenette Dataset with images resized so that the shortest size is 320 px.

name = 'imagenette2-320'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagenette2-320.tgz', '3d9f4d75d012a679600ef8ac0c200d28')]

Imagewoof

class catalyst.contrib.datasets.cv.imagewoof.Imagewoof(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewoof Dataset.

name = 'imagewoof2'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewoof2.tgz', '5eaf5bbf4bf16a77c616dc6e8dd5f8e9')]

class catalyst.contrib.datasets.cv.imagewoof.Imagewoof160(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewoof Dataset with images resized so that the shortest size is 160 px.

name = 'imagewoof2-160'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewoof2-160.tgz', 'fcd23cc7dfce8837c95a8f9d63a128b7')]

class catalyst.contrib.datasets.cv.imagewoof.Imagewoof320(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewoof Dataset with images resized so that the shortest size is 320 px.

name = 'imagewoof2-320'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewoof2-320.tgz', 'af65be7963816efa949fa3c3b4947740')]

Imagewang

class catalyst.contrib.datasets.cv.imagewang.Imagewang(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewang Dataset.

name = 'imagewang'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewang.tgz', '46f9749616a29837e7cd67b103396f6e')]

class catalyst.contrib.datasets.cv.imagewang.Imagewang160(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewang Dataset with images resized so that the shortest size is 160 px.

name = 'imagewang-160'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewang-160.tgz', '1dc388d37d1dc52836c06749e14e37bc')]

class catalyst.contrib.datasets.cv.imagewang.Imagewang320(root: str, train: bool = True, download: bool = False, **kwargs)

Bases: catalyst.contrib.datasets.cv.fastai.ImageClassificationDataset

Imagewang Dataset with images resized so that the shortest size is 320 px.

name = 'imagewang-320'

resources = [('https://s3.amazonaws.com/fast-ai-imageclas/imagewang-320.tgz', 'ff01d7c126230afce776bdf72bda87e6')]

NN

Extensions for torch.nn

Criterion

Cross entropy

class catalyst.contrib.nn.criterion.ce.MaskCrossEntropyLoss(*args, **kwargs)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(*args, **kwargs)

@TODO: Docs. Contribution is welcome.

forward(logits: torch.Tensor, target: torch.Tensor, mask: torch.Tensor) → torch.Tensor

Calculates loss between logits and target tensors.

Parameters

• logits – model logits

• target – true targets

• mask – targets mask

Returns

computed loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.ce.SymmetricCrossEntropyLoss(alpha: float = 1.0, beta: float = 1.0)

Bases: torch.nn.modules.module.Module

The Symmetric Cross Entropy loss.

It has been proposed in Symmetric Cross Entropy for Robust Learning with Noisy Labels.

__init__(alpha: float = 1.0, beta: float = 1.0)

Parameters

• alpha (float) – corresponds to overfitting issue of CE

• beta (float) – corresponds to flexible exploration on the robustness of RCE

forward(input_: torch.Tensor, target: torch.Tensor) → torch.Tensor

Calculates loss between input_ and target tensors.

Parameters

• input_ – input tensor of size (batch_size, num_classes)

• target – target tensor of size (batch_size), where values of a vector correspond to class index

Returns

computed loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.ce.NaiveCrossEntropyLoss(size_average=True)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(size_average=True)

@TODO: Docs. Contribution is welcome.

forward(input_: torch.Tensor, target: torch.Tensor) → torch.Tensor

Calculates loss between input_ and target tensors.

Parameters

• input_ – input tensor of shape …

• target – target tensor of shape …

@TODO: Docs (add shapes). Contribution is welcome.

Circle

class catalyst.contrib.nn.criterion.circle.CircleLoss(margin: float, gamma: float)

Bases: torch.nn.modules.module.Module

CircleLoss from Circle Loss: A Unified Perspective of Pair Similarity Optimization paper.

Adapter from: https://github.com/TinyZeaMays/CircleLoss

Example

>>> import torch
>>> from torch.nn import functional as F
>>> from catalyst.contrib.nn import CircleLoss
>>>
>>> features = F.normalize(torch.rand(256, 64, requires_grad=True))
>>> labels = torch.randint(high=10, size=(256,))
>>> criterion = CircleLoss(margin=0.25, gamma=256)
>>> criterion(features, labels)

__init__(margin: float, gamma: float) → None

Parameters

• margin – margin to use

• gamma – gamma to use

forward(normed_features: torch.Tensor, labels: torch.Tensor) → torch.Tensor

Parameters

• normed_features – batch with samples features of shape [bs; feature_len]

• labels – batch with samples correct labels of shape [bs; ]

Returns

circle loss

Return type

torch.Tensor

Contrastive

class catalyst.contrib.nn.criterion.contrastive.ContrastiveEmbeddingLoss(margin=1.0, reduction='mean')

Bases: torch.nn.modules.module.Module

The Contrastive embedding loss.

It has been proposed in Dimensionality Reduction by Learning an Invariant Mapping.

__init__(margin=1.0, reduction='mean')

Parameters

• margin – margin parameter

• reduction – criterion reduction type

forward(embeddings_left: torch.Tensor, embeddings_right: torch.Tensor, distance_true) → torch.Tensor

Forward propagation method for the contrastive loss.

Parameters

• embeddings_left – left objects embeddings

• embeddings_right – right objects embeddings

• distance_true – true distances

Returns

loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.contrastive.ContrastiveDistanceLoss(margin=1.0, reduction='mean')

Bases: torch.nn.modules.module.Module

The Contrastive distance loss.

@TODO: Docs. Contribution is welcome.

__init__(margin=1.0, reduction='mean')

Parameters

• margin – margin parameter

• reduction – criterion reduction type

forward(distance_pred, distance_true) → torch.Tensor

Forward propagation method for the contrastive loss.

Parameters

• distance_pred – predicted distances

• distance_true – true distances

Returns

loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.contrastive.ContrastivePairwiseEmbeddingLoss(margin=1.0, reduction='mean')

Bases: torch.nn.modules.module.Module

ContrastivePairwiseEmbeddingLoss – proof of concept criterion.

Still work in progress.

@TODO: Docs. Contribution is welcome.

__init__(margin=1.0, reduction='mean')

Parameters

• margin – margin parameter

• reduction – criterion reduction type

forward(embeddings_pred, embeddings_true) → torch.Tensor

Forward propagation method for the contrastive loss.

Work in progress.

Parameters

• embeddings_pred – predicted embeddings

• embeddings_true – true embeddings

Returns

loss

Return type

torch.Tensor

Dice

class catalyst.contrib.nn.criterion.dice.BCEDiceLoss(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid', bce_weight: float = 0.5, dice_weight: float = 0.5)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid', bce_weight: float = 0.5, dice_weight: float = 0.5)

@TODO: Docs. Contribution is welcome.

forward(outputs, targets)

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.criterion.dice.DiceLoss(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

@TODO: Docs. Contribution is welcome.

forward(logits: torch.Tensor, targets: torch.Tensor)

Calculates loss between logits and target tensors.

Parameters

• logits – model logits

• targets – ground truth labels

Returns

computed loss

Focal

class catalyst.contrib.nn.criterion.focal.FocalLossBinary(ignore: int = None, reduced: bool = False, gamma: float = 2.0, alpha: float = 0.25, threshold: float = 0.5, reduction: str = 'mean')

Bases: torch.nn.modules.loss._Loss

Compute focal loss for binary classification problem.

It has been proposed in Focal Loss for Dense Object Detection paper.

@TODO: Docs (add Example). Contribution is welcome.

__init__(ignore: int = None, reduced: bool = False, gamma: float = 2.0, alpha: float = 0.25, threshold: float = 0.5, reduction: str = 'mean')

@TODO: Docs. Contribution is welcome.

forward(logits, targets)

Parameters

• logits – [bs; …]

• targets – [bs; …]

Returns

computed loss

class catalyst.contrib.nn.criterion.focal.FocalLossMultiClass(ignore: int = None, reduced: bool = False, gamma: float = 2.0, alpha: float = 0.25, threshold: float = 0.5, reduction: str = 'mean')

Bases: catalyst.contrib.nn.criterion.focal.FocalLossBinary

Compute focal loss for multi-class problem. Ignores targets having -1 label.

It has been proposed in Focal Loss for Dense Object Detection paper.

@TODO: Docs (add Example). Contribution is welcome.

forward(logits, targets)

Parameters

• logits – [bs; num_classes; …]

• targets – [bs; …]

Returns

computed loss

GAN

class catalyst.contrib.nn.criterion.gan.MeanOutputLoss

Bases: torch.nn.modules.module.Module

Criterion to compute simple mean of the output, completely ignoring target (maybe useful e.g. for WGAN real/fake validity averaging.

forward(output, target)

Compute criterion. @TODO: Docs (add typing). Contribution is welcome.

class catalyst.contrib.nn.criterion.gan.GradientPenaltyLoss

Bases: torch.nn.modules.module.Module

Criterion to compute gradient penalty.

WARN: SHOULD NOT BE RUN WITH CriterionCallback,

use special GradientPenaltyCallback instead

forward(fake_data, real_data, critic, critic_condition_args)

Compute gradient penalty. @TODO: Docs. Contribution is welcome.

Huber

class catalyst.contrib.nn.criterion.huber.HuberLoss(clip_delta=1.0, reduction='mean')

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(clip_delta=1.0, reduction='mean')

@TODO: Docs. Contribution is welcome.

forward(y_pred: torch.Tensor, y_true: torch.Tensor, weights=None) → torch.Tensor

@TODO: Docs. Contribution is welcome.

IOU

class catalyst.contrib.nn.criterion.iou.IoULoss(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Bases: torch.nn.modules.module.Module

The intersection over union (Jaccard) loss.

@TODO: Docs. Contribution is welcome.

__init__(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid')

Parameters

• eps – epsilon to avoid zero division

• threshold – threshold for outputs binarization

• activation – An torch.nn activation applied to the outputs. Must be one of 'none', 'Sigmoid', 'Softmax2d'

forward(outputs, targets)

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.criterion.iou.BCEIoULoss(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid', reduction: str = 'mean')

Bases: torch.nn.modules.module.Module

The Intersection over union (Jaccard) with BCE loss.

@TODO: Docs. Contribution is welcome.

__init__(eps: float = 1e-07, threshold: float = None, activation: str = 'Sigmoid', reduction: str = 'mean')

Parameters

• eps – epsilon to avoid zero division

• threshold – threshold for outputs binarization

• activation – An torch.nn activation applied to the outputs. Must be one of 'none', 'Sigmoid', 'Softmax2d'

• reduction – Specifies the reduction to apply to the output of BCE

forward(outputs, targets)

@TODO: Docs. Contribution is welcome.

Lovasz

class catalyst.contrib.nn.criterion.lovasz.LovaszLossBinary(per_image=False, ignore=None)

Bases: torch.nn.modules.loss._Loss

Creates a criterion that optimizes a binary Lovasz loss.

It has been proposed in The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

__init__(per_image=False, ignore=None)

@TODO: Docs. Contribution is welcome.

forward(logits, targets)

Forward propagation method for the Lovasz loss.

Parameters

• logits – [bs; …]

• targets – [bs; …]

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.criterion.lovasz.LovaszLossMultiClass(per_image=False, ignore=None)

Bases: torch.nn.modules.loss._Loss

Creates a criterion that optimizes a multi-class Lovasz loss.

It has been proposed in The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

__init__(per_image=False, ignore=None)

@TODO: Docs. Contribution is welcome.

forward(logits, targets)

Forward propagation method for the Lovasz loss.

Parameters

• logits – [bs; num_classes; …]

• targets – [bs; …]

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.criterion.lovasz.LovaszLossMultiLabel(per_image=False, ignore=None)

Bases: torch.nn.modules.loss._Loss

Creates a criterion that optimizes a multi-label Lovasz loss.

It has been proposed in The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

__init__(per_image=False, ignore=None)

@TODO: Docs. Contribution is welcome.

forward(logits, targets)

Forward propagation method for the Lovasz loss.

Parameters

• logits – [bs; num_classes; …]

• targets – [bs; num_classes; …]

@TODO: Docs. Contribution is welcome.

Margin

class catalyst.contrib.nn.criterion.margin.MarginLoss(alpha: float = 0.2, beta: float = 1.0, skip_labels: Union[int, List[int]] = -1)

Bases: torch.nn.modules.module.Module

Margin loss criterion

__init__(alpha: float = 0.2, beta: float = 1.0, skip_labels: Union[int, List[int]] = -1)

Margin loss constructor.

Parameters

• alpha – alpha

• beta – beta

• skip_labels (int or List[int]) – labels to skip

forward(embeddings: torch.Tensor, targets: torch.Tensor) → torch.Tensor

Forward method for the margin loss.

Parameters

• embeddings – tensor with embeddings

• targets – tensor with target labels

Returns

computed loss

Triplet

class catalyst.contrib.nn.criterion.triplet.TripletLoss(margin: float = 0.3)

Bases: torch.nn.modules.module.Module

Triplet loss with hard positive/negative mining.

Adapted from: https://github.com/NegatioN/OnlineMiningTripletLoss

__init__(margin: float = 0.3)

Parameters

margin – margin for triplet

forward(embeddings, targets)

Forward propagation method for the triplet loss.

Parameters

• embeddings – tensor of shape (batch_size, embed_dim)

• targets – labels of the batch, of size (batch_size,)

Returns

scalar tensor containing the triplet loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.triplet.TripletLossV2(margin=0.3)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(margin=0.3)

Parameters

margin – margin for triplet.

forward(embeddings, targets)

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.criterion.triplet.TripletPairwiseEmbeddingLoss(margin: float = 0.3, reduction: str = 'mean')

Bases: torch.nn.modules.module.Module

TripletPairwiseEmbeddingLoss – proof of concept criterion.

Still work in progress.

@TODO: Docs. Contribution is welcome.

__init__(margin: float = 0.3, reduction: str = 'mean')

Parameters

• margin – margin parameter

• reduction – criterion reduction type

forward(embeddings_pred, embeddings_true)

Work in progress.

Parameters

• embeddings_pred – predicted embeddings with shape [batch_size, embedding_size]

• embeddings_true – true embeddings with shape [batch_size, embedding_size]

Returns

loss

Return type

torch.Tensor

class catalyst.contrib.nn.criterion.triplet.TripletMarginLossWithSampler(margin: float, sampler_inbatch: IInbatchTripletSampler)

Bases: torch.nn.modules.module.Module

This class combines in-batch sampling of triplets and default TripletMargingLoss from PyTorch.

__init__(margin: float, sampler_inbatch: IInbatchTripletSampler)

Parameters

• margin – margin value

• sampler_inbatch – sampler for forming triplets inside the batch

forward(features: torch.Tensor, labels: Union[torch.Tensor, List[int]]) → torch.Tensor

Parameters

• features – features with the shape of [batch_size, features_dim]

• labels – labels of samples having batch_size elements

Returns: loss value

Wing

class catalyst.contrib.nn.criterion.wing.WingLoss(width: int = 5, curvature: float = 0.5, reduction: str = 'mean')

Bases: torch.nn.modules.module.Module

Creates a criterion that optimizes a Wing loss.

It has been proposed in Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks.

Examples

@TODO: Docs. Contribution is welcome.

Adapted from: https://github.com/BloodAxe/pytorch-toolbelt

__init__(width: int = 5, curvature: float = 0.5, reduction: str = 'mean')

Parameters

@TODO – Docs. Contribution is welcome.

forward(outputs: torch.Tensor, targets: torch.Tensor) → torch.Tensor

Parameters

@TODO – Docs. Contribution is welcome.

Modules

ArcFace and SubCenterArcFace

class catalyst.contrib.nn.modules.arcface.ArcFace(in_features: int, out_features: int, s: float = 64.0, m: float = 0.5, eps: float = 1e-06)

Bases: torch.nn.modules.module.Module

Implementation of ArcFace: Additive Angular Margin Loss for Deep Face Recognition.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

• s – norm of input feature. Default: 64.0.

• m – margin. Default: 0.5.

• eps – operation accuracy. Default: 1e-6.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = ArcFace(5, 10, s=1.31, m=0.5)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding, target)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor, target: torch.LongTensor) → torch.Tensor

Parameters

• input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

• target – target classes, expected shapes B where B is batch dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes (out_features).

class catalyst.contrib.nn.modules.arcface.SubCenterArcFace(in_features: int, out_features: int, s: float = 64.0, m: float = 0.5, k: int = 3, eps: float = 1e-06)

Bases: torch.nn.modules.module.Module

Implementation of Sub-center ArcFace: Boosting Face Recognition by Large-scale Noisy Web Faces.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

• s – norm of input feature, Default: 64.0.

• m – margin. Default: 0.5.

• k – number of possible class centroids. Default: 3.

• eps (float, optional) – operation accuracy. Default: 1e-6.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = SubCenterArcFace(5, 10, s=1.31, m=0.35, k=2)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding, target)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor, label: torch.LongTensor) → torch.Tensor

Parameters

• input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

• label – target classes, expected shapes B where B is batch dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes.

Arc Margin Product

class catalyst.contrib.nn.modules.arcmargin.ArcMarginProduct(in_features: int, out_features: int)

Bases: torch.nn.modules.module.Module

Implementation of Arc Margin Product.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = ArcMarginProduct(5, 10)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor) → torch.Tensor

Parameters

input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes (out_features).

Common modules

class catalyst.contrib.nn.modules.common.Flatten

Bases: torch.nn.modules.module.Module

Flattens the input. Does not affect the batch size.

@TODO: Docs (add Example). Contribution is welcome.

__init__()

@TODO: Docs. Contribution is welcome.

forward(x)

Forward call.

class catalyst.contrib.nn.modules.common.Lambda(lambda_fn)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(lambda_fn)

@TODO: Docs. Contribution is welcome.

forward(x)

Forward call.

class catalyst.contrib.nn.modules.common.Normalize(**normalize_kwargs)

Bases: torch.nn.modules.module.Module

Performs \(L_p\) normalization of inputs over specified dimension.

@TODO: Docs (add Example). Contribution is welcome.

__init__(**normalize_kwargs)

Parameters

**normalize_kwargs – see torch.nn.functional.normalize params

forward(x)

Forward call.

class catalyst.contrib.nn.modules.common.GaussianNoise(stddev: float = 0.1)

Bases: torch.nn.modules.module.Module

A gaussian noise module.

Shape:

• Input: (batch, *)

• Output: (batch, *) (same shape as input)

__init__(stddev: float = 0.1)

Parameters

stddev – The standard deviation of the normal distribution. Default: 0.1.

forward(x: torch.Tensor)

Forward call.

CosFace and AdaCos

class catalyst.contrib.nn.modules.cosface.CosFace(in_features: int, out_features: int, s: float = 64.0, m: float = 0.35)

Bases: torch.nn.modules.module.Module

Implementation of CosFace: Large Margin Cosine Loss for Deep Face Recognition.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

• s – norm of input feature. Default: 64.0.

• m – margin. Default: 0.35.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = CosFaceLoss(5, 10, s=1.31, m=0.1)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding, target)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor, target: torch.LongTensor) → torch.Tensor

Parameters

• input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

• target – target classes, expected shapes B where B is batch dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes (out_features).

class catalyst.contrib.nn.modules.cosface.AdaCos(in_features: int, out_features: int, dynamical_s: bool = True, eps: float = 1e-06)

Bases: torch.nn.modules.module.Module

Implementation of AdaCos: Adaptively Scaling Cosine Logits for Effectively Learning Deep Face Representations.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

• dynamical_s – option to use dynamical scale parameter. If False then will be used initial scale. Default: True.

• eps – operation accuracy. Default: 1e-6.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = AdaCos(5, 10)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding, target)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor, target: torch.LongTensor) → torch.Tensor

Parameters

• input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

• target – target classes, expected shapes B where B is batch dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes (out_features).

Last-Mean-Average-Attention (LAMA)-Pooling

class catalyst.contrib.nn.modules.lama.TemporalLastPooling

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Forward call.

class catalyst.contrib.nn.modules.lama.TemporalAvgPooling

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Forward call.

class catalyst.contrib.nn.modules.lama.TemporalMaxPooling

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Forward call.

class catalyst.contrib.nn.modules.lama.TemporalDropLastWrapper(net)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(net)

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None)

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.nn.modules.lama.TemporalAttentionPooling(in_features, activation=None, kernel_size=1, **params)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(in_features, activation=None, kernel_size=1, **params)

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Forward call.

Parameters

• x – tensor of size (batch_size, history_len, feature_size)

• mask – mask to use

Returns

pooling result

name2activation = {'sigmoid': Sigmoid(), 'softmax': Softmax(dim=1), 'tanh': Tanh()}

class catalyst.contrib.nn.modules.lama.TemporalConcatPooling(in_features, history_len=1)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(in_features, history_len=1)

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Concat pooling forward.

Parameters

• x – tensor of size (batch_size, history_len, feature_size)

• mask – mask to use

Returns

concated result

class catalyst.contrib.nn.modules.lama.LamaPooling(in_features, groups=None)

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(in_features, groups=None)

@TODO: Docs. Contribution is welcome.

available_groups = ['last', 'avg', 'avg_droplast', 'max', 'max_droplast', 'sigmoid', 'sigmoid_droplast', 'softmax', 'softmax_droplast', 'tanh', 'tanh_droplast']

forward(x: torch.Tensor, mask: torch.Tensor = None) → torch.Tensor

Forward method of the LAMA.

Parameters

• x – tensor of size (batch_size, history_len, feature_size)

• mask – mask to use for attention compute

Returns

LAMA pooling result

Return type

torch.Tensor

Pooling

class catalyst.contrib.nn.modules.pooling.GlobalAttnPool2d(in_features, activation_fn='Sigmoid')

Bases: torch.nn.modules.module.Module

@TODO: Docs. Contribution is welcome.

__init__(in_features, activation_fn='Sigmoid')

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalAvgAttnPool2d(in_features, activation_fn='Sigmoid')

Bases: torch.nn.modules.module.Module

@TODO: Docs (add Example). Contribution is welcome.

__init__(in_features, activation_fn='Sigmoid')

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalAvgPool2d

Bases: torch.nn.modules.module.Module

Applies a 2D global average pooling operation over an input signal composed of several input planes.

@TODO: Docs (add Example). Contribution is welcome.

__init__()

Constructor method for the GlobalAvgPool2d class.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalConcatAttnPool2d(in_features, activation_fn='Sigmoid')

Bases: torch.nn.modules.module.Module

@TODO: Docs (add Example). Contribution is welcome.

__init__(in_features, activation_fn='Sigmoid')

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalConcatPool2d

Bases: torch.nn.modules.module.Module

@TODO: Docs (add Example). Contribution is welcome.

__init__()

Constructor method for the GlobalConcatPool2d class.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalMaxAttnPool2d(in_features, activation_fn='Sigmoid')

Bases: torch.nn.modules.module.Module

@TODO: Docs (add Example). Contribution is welcome.

__init__(in_features, activation_fn='Sigmoid')

@TODO: Docs. Contribution is welcome.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

class catalyst.contrib.nn.modules.pooling.GlobalMaxPool2d

Bases: torch.nn.modules.module.Module

Applies a 2D global max pooling operation over an input signal composed of several input planes.

@TODO: Docs (add Example). Contribution is welcome.

__init__()

Constructor method for the GlobalMaxPool2d class.

forward(x: torch.Tensor) → torch.Tensor

Forward call.

static out_features(in_features)

Returns number of channels produced by the pooling.

Parameters

in_features – number of channels in the input sample

Returns

number of output features

RMSNorm

class catalyst.contrib.nn.modules.rms_norm.RMSNorm(dimension: int, epsilon: float = 1e-08, is_bias: bool = False)

Bases: torch.nn.modules.module.Module

An implementation of RMS Normalization.

@TODO: Docs (link to paper). Contribution is welcome.

__init__(dimension: int, epsilon: float = 1e-08, is_bias: bool = False)

Parameters

• dimension – the dimension of the layer output to normalize

• epsilon – an epsilon to prevent dividing by zero in case the layer has zero variance. (default = 1e-8)

• is_bias – a boolean value whether to include bias term while normalization

forward(x: torch.Tensor) → torch.Tensor

@TODO: Docs. Contribution is welcome.

SqueezeAndExcitation

class catalyst.contrib.nn.modules.se.sSE(in_channels: int)

Bases: torch.nn.modules.module.Module

The sSE (Channel Squeeze and Spatial Excitation) block from the Concurrent Spatial and Channel ‘Squeeze & Excitation’ in Fully Convolutional Networks paper.

Adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178

Shape:

• Input: (batch, channels, height, width)

• Output: (batch, channels, height, width) (same shape as input)

__init__(in_channels: int)

Parameters

in_channels – The number of channels in the feature map of the input.

forward(x: torch.Tensor)

Forward call.

class catalyst.contrib.nn.modules.se.scSE(in_channels: int, r: int = 16)

Bases: torch.nn.modules.module.Module

The scSE (Concurrent Spatial and Channel Squeeze and Channel Excitation) block from the Concurrent Spatial and Channel ‘Squeeze & Excitation’ in Fully Convolutional Networks paper.

Adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178

Shape:

• Input: (batch, channels, height, width)

• Output: (batch, channels, height, width) (same shape as input)

__init__(in_channels: int, r: int = 16)

Parameters

• in_channels – The number of channels in the feature map of the input.

• r – The reduction ratio of the intermediate channels. Default: 16.

forward(x: torch.Tensor)

Forward call.

class catalyst.contrib.nn.modules.se.cSE(in_channels: int, r: int = 16)

Bases: torch.nn.modules.module.Module

The channel-wise SE (Squeeze and Excitation) block from the Squeeze-and-Excitation Networks paper.

Adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65939 and https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178

Shape:

• Input: (batch, channels, height, width)

• Output: (batch, channels, height, width) (same shape as input)

__init__(in_channels: int, r: int = 16)

Parameters

• in_channels – The number of channels in the feature map of the input.

• r – The reduction ratio of the intermediate channels. Default: 16.

forward(x: torch.Tensor)

Forward call.

SoftMax

class catalyst.contrib.nn.modules.softmax.SoftMax(in_features: int, num_classes: int)

Bases: torch.nn.modules.module.Module

Implementation of Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features.

Parameters

• in_features – size of each input sample.

• out_features – size of each output sample.

Shape:

• Input: \((batch, H_{in})\) where \(H_{in} = in\_features\).

• Output: \((batch, H_{out})\) where \(H_{out} = out\_features\).

Example

>>> layer = SoftMax(5, 10)
>>> loss_fn = nn.CrosEntropyLoss()
>>> embedding = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(10)
>>> output = layer(embedding, target)
>>> loss = loss_fn(output, target)
>>> loss.backward()

forward(input: torch.Tensor) → torch.Tensor

Parameters

input – input features, expected shapes BxF where B is batch dimension and F is an input feature dimension.

Returns

tensor (logits) with shapes BxC where C is a number of classes (out_features).

Optimizers

AdamP

AdamP Copyright (c) 2020-present NAVER Corp. MIT license

Original source code: https://github.com/clovaai/AdamP

class catalyst.contrib.nn.optimizers.adamp.AdamP(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, delta=0.1, wd_ratio=0.1, nesterov=False)

Bases: torch.optim.optimizer.Optimizer

Implements AdamP algorithm.

The original Adam algorithm was proposed in Adam: A Method for Stochastic Optimization. The AdamP variant was proposed in Slowing Down the Weight Norm Increase in Momentum-based Optimizers.

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr (float, optional) – learning rate (default: 1e-3)

• betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• weight_decay (float, optional) – weight decay coefficient (default: 0)

• delta – threshold that determines whether a set of parameters is scale invariant or not (default: 0.1)

• wd_ratio – relative weight decay applied on scale-invariant parameters compared to that applied on scale-variant parameters (default: 0.1)

• nesterov (boolean, optional) – enables Nesterov momentum (default: False)

__init__(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, delta=0.1, wd_ratio=0.1, nesterov=False)

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr (float, optional) – learning rate (default: 1e-3)

• betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• weight_decay (float, optional) – weight decay coefficient (default: 1e-2)

• delta – threshold that determines whether a set of parameters is scale invariant or not (default: 0.1)

• wd_ratio – relative weight decay applied on scale-invariant parameters compared to that applied on scale-variant parameters (default: 0.1)

• nesterov (boolean, optional) – enables Nesterov momentum (default: False)

step(closure=None)

Performs a single optimization step (parameter update).

Parameters

closure – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

Returns

computed loss

Lamb

class catalyst.contrib.nn.optimizers.lamb.Lamb(params, lr: Optional[float] = 0.001, betas: Optional[Tuple[float, float]] = (0.9, 0.999), eps: Optional[float] = 1e-06, weight_decay: Optional[float] = 0.0, adam: Optional[bool] = False)

Bases: torch.optim.optimizer.Optimizer

Implements Lamb algorithm.

It has been proposed in Training BERT in 76 minutes.

__init__(params, lr: Optional[float] = 0.001, betas: Optional[Tuple[float, float]] = (0.9, 0.999), eps: Optional[float] = 1e-06, weight_decay: Optional[float] = 0.0, adam: Optional[bool] = False)

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr (float, optional) – learning rate (default: 1e-3)

• betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

• adam (bool, optional) – always use trust ratio = 1, which turns this into Adam. Useful for comparison purposes.

step(closure: Optional[Callable] = None)

Makes optimizer step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

Returns

computed loss

Raises

RuntimeError – Lamb does not support sparse gradients

Lookahead

class catalyst.contrib.nn.optimizers.lookahead.Lookahead(optimizer: torch.optim.optimizer.Optimizer, k: int = 5, alpha: float = 0.5)

Bases: torch.optim.optimizer.Optimizer

Implements Lookahead algorithm.

It has been proposed in Lookahead Optimizer: k steps forward, 1 step back.

Adapted from: https://github.com/alphadl/lookahead.pytorch (MIT License)

__init__(optimizer: torch.optim.optimizer.Optimizer, k: int = 5, alpha: float = 0.5)

@TODO: Docs. Contribution is welcome.

add_param_group(param_group)

@TODO: Docs. Contribution is welcome.

classmethod get_from_params(params: Dict, base_optimizer_params: Dict = None, **kwargs) → catalyst.contrib.nn.optimizers.lookahead.Lookahead

@TODO: Docs. Contribution is welcome.

load_state_dict(state_dict)

@TODO: Docs. Contribution is welcome.

state_dict()

@TODO: Docs. Contribution is welcome.

step(closure: Optional[Callable] = None)

Makes optimizer step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

Returns

computed loss

update(group)

@TODO: Docs. Contribution is welcome.

update_lookahead()

@TODO: Docs. Contribution is welcome.

QHAdamW

class catalyst.contrib.nn.optimizers.qhadamw.QHAdamW(params, lr=0.001, betas=(0.995, 0.999), nus=(0.7, 1.0), weight_decay=0.0, eps=1e-08)

Bases: torch.optim.optimizer.Optimizer

Implements QHAdam algorithm.

Combines QHAdam algorithm that was proposed in Quasi-hyperbolic momentum and Adam for deep learning with weight decay decoupling from Decoupled Weight Decay Regularization paper.

Example

>>> optimizer = QHAdamW(
...     model.parameters(),
...     lr=3e-4, nus=(0.8, 1.0), betas=(0.99, 0.999))
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> optimizer.step()

Adapted from: https://github.com/iprally/qhadamw-pytorch/blob/master/qhadamw.py (MIT License)

__init__(params, lr=0.001, betas=(0.995, 0.999), nus=(0.7, 1.0), weight_decay=0.0, eps=1e-08)

Parameters

• params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

• lr (float, optional) – learning rate (\(\alpha\) from the paper) (default: 1e-3)

• betas (Tuple[float, float], optional) – coefficients used for computing running averages of the gradient and its square (default: (0.995, 0.999))

• nus (Tuple[float, float], optional) – immediate discount factors used to estimate the gradient and its square (default: (0.7, 1.0))

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• weight_decay (float, optional) – weight decay (L2 regularization coefficient, times two) (default: 0.0)

step(closure: Optional[Callable] = None)

Makes optimizer step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

Returns

computed loss

Raises

RuntimeError – QHAdamW does not support sparse gradients

RAdam

class catalyst.contrib.nn.optimizers.radam.RAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)

Bases: torch.optim.optimizer.Optimizer

Implements RAdam algorithm.

It has been proposed in On the Variance of the Adaptive Learning Rate and Beyond.

@TODO: Docs (add Example). Contribution is welcome

Adapted from: https://github.com/LiyuanLucasLiu/RAdam (Apache-2.0 License)

__init__(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)

@TODO: Docs. Contribution is welcome.

step(closure: Optional[Callable] = None)

Makes optimizer step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

Returns

computed loss

Raises

RuntimeError – RAdam does not support sparse gradients

Ralamb

class catalyst.contrib.nn.optimizers.ralamb.Ralamb(params: Iterable, lr: float = 0.001, betas: Tuple[float, float] = (0.9, 0.999), eps: float = 1e-08, weight_decay: float = 0)

Bases: torch.optim.optimizer.Optimizer

RAdam optimizer with LARS/LAMB tricks.

Adapted from: https://github.com/mgrankin/over9000/blob/master/ralamb.py (Apache-2.0 License)

__init__(params: Iterable, lr: float = 0.001, betas: Tuple[float, float] = (0.9, 0.999), eps: float = 1e-08, weight_decay: float = 0)

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr (float, optional) – learning rate (default: 1e-3)

• betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

step(closure: Optional[Callable] = None)

Makes optimizer step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

Returns

computed loss

Raises

RuntimeError – Ralamb does not support sparse gradients

SGDP

AdamP Copyright (c) 2020-present NAVER Corp. MIT license

Original source code: https://github.com/clovaai/AdamP

class catalyst.contrib.nn.optimizers.sgdp.SGDP(params, lr=<required parameter>, momentum=0, weight_decay=0, dampening=0, nesterov=False, eps=1e-08, delta=0.1, wd_ratio=0.1)

Bases: torch.optim.optimizer.Optimizer

Implements SGDP algorithm.

The SGDP variant was proposed in Slowing Down the Weight Norm Increase in Momentum-based Optimizers.

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr – learning rate

• momentum (float, optional) – momentum factor (default: 0)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

• dampening (float, optional) – dampening for momentum (default: 0)

• nesterov (bool, optional) – enables Nesterov momentum (default: False)

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• delta – threshold that determines whether a set of parameters is scale invariant or not (default: 0.1)

• wd_ratio – relative weight decay applied on scale-invariant parameters compared to that applied on scale-variant parameters (default: 0.1)

__init__(params, lr=<required parameter>, momentum=0, weight_decay=0, dampening=0, nesterov=False, eps=1e-08, delta=0.1, wd_ratio=0.1)

Parameters

• params – iterable of parameters to optimize or dicts defining parameter groups

• lr – learning rate

• momentum (float, optional) – momentum factor (default: 0)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

• dampening (float, optional) – dampening for momentum (default: 0)

• nesterov (bool, optional) – enables Nesterov momentum (default: False)

• eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

• delta – threshold that determines whether a set of parameters is scale invariant or not (default: 0.1)

• wd_ratio – relative weight decay applied on scale-invariant parameters compared to that applied on scale-variant parameters (default: 0.1)

step(closure=None)

Performs a single optimization step (parameter update).

Parameters

closure – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

Returns

computed loss

Schedulers

class catalyst.contrib.nn.schedulers.base.BaseScheduler(optimizer, last_epoch=-1)

Bases: torch.optim.lr_scheduler._LRScheduler, abc.ABC

Base class for all schedulers with momentum update.

abstract get_momentum() → List[float]

Function that returns the new momentum for optimizer.

step(epoch: Optional[int] = None) → None

Make one scheduler step.

Parameters

epoch (int, optional) – current epoch num

class catalyst.contrib.nn.schedulers.base.BatchScheduler(optimizer, last_epoch=-1)

Bases: catalyst.contrib.nn.schedulers.base.BaseScheduler, abc.ABC

@TODO: Docs. Contribution is welcome.

OneCycleLRWithWarmup

class catalyst.contrib.nn.schedulers.onecycle.OneCycleLRWithWarmup(optimizer: torch.optim.optimizer.Optimizer, num_steps: int, lr_range=(1.0, 0.005), init_lr: float = None, warmup_steps: int = 0, warmup_fraction: float = None, decay_steps: int = 0, decay_fraction: float = None, momentum_range=(0.8, 0.99, 0.999), init_momentum: float = None)

Bases: catalyst.contrib.nn.schedulers.base.BatchScheduler

OneCycle scheduler with warm-up & lr decay stages.

First stage increases lr from init_lr to max_lr, and called warmup. Also it decreases momentum from init_momentum to min_momentum. Takes warmup_steps steps

Second is annealing stage. Decrease lr from max_lr to min_lr, Increase momentum from min_momentum to max_momentum.

Third, optional, lr decay.

__init__(optimizer: torch.optim.optimizer.Optimizer, num_steps: int, lr_range=(1.0, 0.005), init_lr: float = None, warmup_steps: int = 0, warmup_fraction: float = None, decay_steps: int = 0, decay_fraction: float = None, momentum_range=(0.8, 0.99, 0.999), init_momentum: float = None)

Parameters

• optimizer – PyTorch optimizer

• num_steps – total number of steps

• lr_range – tuple with two or three elements (max_lr, min_lr, [final_lr])

• init_lr (float, optional) – initial lr

• warmup_steps – count of steps for warm-up stage

• warmup_fraction (float, optional) – fraction in [0; 1) to calculate number of warmup steps. Cannot be set together with warmup_steps

• decay_steps – count of steps for lr decay stage

• decay_fraction (float, optional) – fraction in [0; 1) to calculate number of decay steps. Cannot be set together with decay_steps

• momentum_range – tuple with two or three elements (min_momentum, max_momentum, [final_momentum])

• init_momentum (float, optional) – initial momentum

get_lr() → List[float]

Function that returns the new lr for optimizer.

Returns

calculated lr for every param groups

Return type

List[float]

get_momentum() → List[float]

Function that returns the new momentum for optimizer.

Returns

calculated momentum for every param groups

Return type

List[float]

recalculate(loader_len: int, current_step: int) → None

Recalculates total num_steps for batch mode.

Parameters

• loader_len – total count of batches in an epoch

• current_step – current step

reset()

@TODO: Docs. Contribution is welcome.

Models

Segmentation

Unet

class catalyst.contrib.models.cv.segmentation.unet.Unet(num_classes: int = 1, in_channels: int = 3, num_channels: int = 32, num_blocks: int = 4, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.UnetSpec

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.models.cv.segmentation.unet.ResnetUnet(num_classes: int = 1, arch: str = 'resnet18', pretrained: bool = True, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.ResnetUnetSpec

@TODO: Docs. Contribution is welcome.

Linknet

class catalyst.contrib.models.cv.segmentation.linknet.Linknet(num_classes: int = 1, in_channels: int = 3, num_channels: int = 32, num_blocks: int = 4, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.UnetSpec

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.models.cv.segmentation.linknet.ResnetLinknet(num_classes: int = 1, arch: str = 'resnet18', pretrained: bool = True, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.ResnetUnetSpec

@TODO: Docs. Contribution is welcome.

FPNnet

class catalyst.contrib.models.cv.segmentation.fpn.FPNUnet(num_classes: int = 1, in_channels: int = 3, num_channels: int = 32, num_blocks: int = 4, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.UnetSpec

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.models.cv.segmentation.fpn.ResnetFPNUnet(num_classes: int = 1, arch: str = 'resnet18', pretrained: bool = True, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.ResnetUnetSpec

@TODO: Docs. Contribution is welcome.

PSPnet

class catalyst.contrib.models.cv.segmentation.psp.PSPnet(num_classes: int = 1, in_channels: int = 3, num_channels: int = 32, num_blocks: int = 4, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.UnetSpec

@TODO: Docs. Contribution is welcome.

class catalyst.contrib.models.cv.segmentation.psp.ResnetPSPnet(num_classes: int = 1, arch: str = 'resnet18', pretrained: bool = True, encoder_params: Dict = None, bridge_params: Dict = None, decoder_params: Dict = None, head_params: Dict = None, state_dict: Union[dict, str, pathlib.Path] = None)

Bases: catalyst.contrib.models.cv.segmentation.core.ResnetUnetSpec

@TODO: Docs. Contribution is welcome.