Metrics¶
Accuracy¶
 Various accuracy metrics:

catalyst.metrics.accuracy.
accuracy
(outputs: torch.Tensor, targets: torch.Tensor, topk: Sequence[int] = (1, ), activation: Optional[str] = None) → Sequence[torch.Tensor][source]¶ Computes multiclass accuracy@topk for the specified values of topk.
 Parameters
outputs – model outputs, logits with shape [bs; num_classes]
targets – ground truth, labels with shape [bs; 1]
activation – activation to use for model output
topk – topk for accuracy@topk computing
 Returns
list with computed accuracy@topk

catalyst.metrics.accuracy.
multi_label_accuracy
(outputs: torch.Tensor, targets: torch.Tensor, threshold: Union[float, torch.Tensor], activation: Optional[str] = None) → torch.Tensor[source]¶ Computes multilabel accuracy for the specified activation and threshold.
 Parameters
outputs – NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.
targets – binary NxK tensort that encodes which of the K classes are associated with the Nth input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)
threshold – threshold for for model output
activation – activation to use for model output
 Returns
computed multilabel accuracy
AUC¶

catalyst.metrics.auc.
auc
(outputs: torch.Tensor, targets: torch.Tensor) → torch.Tensor[source]¶ AUC metric.
 Parameters
outputs – [bs; num_classes] estimated scores from a model.
targets – [bs; num_classes] ground truth (correct) target values.
 Returns
Tensor with [num_classes] shape of perclassaucs
 Return type
torch.Tensor
CMC score¶

catalyst.metrics.cmc_score.
cmc_score_count
(distances: torch.Tensor, conformity_matrix: torch.Tensor, topk: int = 1) → float[source]¶ Function to count CMC from distance matrix and conformity matrix.
 Parameters
distances – distance matrix shape of (n_embeddings_x, n_embeddings_y)
conformity_matrix – binary matrix with 1 on same label pos and 0 otherwise
topk – number of top examples for cumulative score counting
 Returns
cmc score

catalyst.metrics.cmc_score.
cmc_score
(query_embeddings: torch.Tensor, gallery_embeddings: torch.Tensor, conformity_matrix: torch.Tensor, topk: int = 1) → float[source]¶ Function to count CMC score from query and gallery embeddings.
 Parameters
query_embeddings – tensor shape of (n_embeddings, embedding_dim) embeddings of the objects in querry
gallery_embeddings – tensor shape of (n_embeddings, embedding_dim) embeddings of the objects in gallery
conformity_matrix – binary matrix with 1 on same label pos and 0 otherwise
topk – number of top examples for cumulative score counting
 Returns
cmc score
Dice¶
Dice metric.

catalyst.metrics.dice.
dice
(outputs: torch.Tensor, targets: torch.Tensor, eps: float = 1e07, threshold: float = None, activation: str = 'Sigmoid')[source]¶ Computes the dice metric.
 Parameters
outputs – a list of predicted elements
targets – a list of elements that are to be predicted
eps – epsilon
threshold – threshold for outputs binarization
activation – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]
 Returns
Dice score
 Return type
float

catalyst.metrics.dice.
calculate_dice
(true_positives: numpy.array, false_positives: numpy.array, false_negatives: numpy.array) → numpy.array[source]¶ Calculate list of Dice coefficients.
 Parameters
true_positives – true positives numpy tensor
false_positives – false positives numpy tensor
false_negatives – false negatives numpy tensor
 Returns
dice score
 Return type
np.array
 Raises
ValueError – if dice is out of [0; 1] bounds
F1 score¶
F1 score.

catalyst.metrics.f1_score.
f1_score
(outputs: torch.Tensor, targets: torch.Tensor, beta: float = 1.0, eps: float = 1e07, threshold: float = None, activation: str = 'Sigmoid')[source]¶  Parameters
outputs – A list of predicted elements
targets – A list of elements that are to be predicted
eps – epsilon to avoid zero division
beta – beta param for f_score
threshold – threshold for outputs binarization
activation – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]
 Returns
F_1 score
 Return type
float
Focal¶
 Focal losses:

catalyst.metrics.focal.
sigmoid_focal_loss
(outputs: torch.Tensor, targets: torch.Tensor, gamma: float = 2.0, alpha: float = 0.25, reduction: str = 'mean')[source]¶ Compute binary focal loss between target and output logits.
 Parameters
outputs – tensor of arbitrary shape
targets – tensor of the same shape as input
gamma – gamma for focal loss
alpha – alpha for focal loss
reduction (string, optional) – specifies the reduction to apply to the output:
"none"
"mean"
"sum"
"batchwise_mean"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed.
 Returns
computed loss

catalyst.metrics.focal.
reduced_focal_loss
(outputs: torch.Tensor, targets: torch.Tensor, threshold: float = 0.5, gamma: float = 2.0, reduction='mean') → torch.Tensor[source]¶ Compute reduced focal loss between target and output logits.
It has been proposed in Reduced Focal Loss: 1st Place Solution to xView object detection in Satellite Imagery paper.
Note
size_average
andreduce
params are in the process of being deprecated, and in the meantime, specifying either of those two args will overridereduction
.Source: https://github.com/BloodAxe/pytorchtoolbelt
 Parameters
outputs – tensor of arbitrary shape
targets – tensor of the same shape as input
threshold – threshold for focal reduction
gamma – gamma for focal reduction
reduction (string, optional) – specifies the reduction to apply to the output:
"none"
"mean"
"sum"
"batchwise_mean"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed."batchwise_mean"
computes mean loss per sample in batch. Default: “mean”
 Returns: # noqa: DAR201
torch.Tensor: computed loss
IoU¶
IoU metric. Jaccard metric refers to IoU here, same functionality.

catalyst.metrics.iou.
iou
(outputs: torch.Tensor, targets: torch.Tensor, classes: List[str] = None, eps: float = 1e07, threshold: float = None, activation: str = 'Sigmoid') → torch.Tensor[source]¶  Parameters
outputs – A list of predicted elements
targets – A list of elements that are to be predicted
classes – if classes are specified we reduce across all dims except channels
eps – epsilon to avoid zero division
threshold – threshold for outputs binarization
activation – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]
 Returns
IoU (Jaccard) score(s)
 Return type
Union[float, List[float]]

catalyst.metrics.iou.
jaccard
(outputs: torch.Tensor, targets: torch.Tensor, classes: List[str] = None, eps: float = 1e07, threshold: float = None, activation: str = 'Sigmoid') → torch.Tensor¶  Parameters
outputs – A list of predicted elements
targets – A list of elements that are to be predicted
classes – if classes are specified we reduce across all dims except channels
eps – epsilon to avoid zero division
threshold – threshold for outputs binarization
activation – An torch.nn activation applied to the outputs. Must be one of [“none”, “Sigmoid”, “Softmax2d”]
 Returns
IoU (Jaccard) score(s)
 Return type
Union[float, List[float]]
MRR¶
MRR metric.

catalyst.metrics.mrr.
mrr
(outputs: torch.Tensor, targets: torch.Tensor, k=100) → torch.Tensor[source]¶ Calculate the Mean Reciprocal Rank (MRR) score given model ouptputs and targets User’s data aggreagtesd in batches.
The MRR@k is the mean overall user of the reciprocal rank, that is the rank of the highest ranked relevant item, if any in the top k, 0 otherwise. https://en.wikipedia.org/wiki/Mean_reciprocal_rank
 Parameters
outputs (torch.Tensor) – Tensor weith predicted score size: [batch_size, slate_length] model outputs, logits
targets (torch.Tensor) – Binary tensor with ground truth. 1 means the item is relevant for the user and 0 not relevant size: [batch_szie, slate_length] ground truth, labels
k (int) – Parameter fro evaluation on topk items
 Returns
The mrr score for each user.
 Return type
result (torch.Tensor)
Precision¶

catalyst.metrics.precision.
average_precision
(outputs: torch.Tensor, targets: torch.Tensor, weights: Optional[torch.Tensor] = None) → torch.Tensor[source]¶ Computes the average precision.
 Parameters
outputs – NxK tensor that for each of the N examples indicates the probability of the example belonging to each of the K classes, according to the model.
targets – binary NxK tensort that encodes which of the K classes are associated with the Nth input (eg: a row [0, 1, 0, 1] indicates that the example is associated with classes 2 and 4)
weights – importance for each sample
 Returns
tensor of [K; ] shape, with average precision for K classes
 Return type
torch.Tensor
Functional¶

catalyst.metrics.functional.
get_binary_statistics
(predictions: torch.Tensor, targets: torch.Tensor) → Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor][source]¶ Computes the number of true positive, false positive, true negative, false negative and support for a binary classification problem.
 Parameters
predictions – Estimated targets as predicted by a model.
targets – Ground truth (correct) target values.
 Returns
stats
 Return type
Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]

catalyst.metrics.functional.
wrap_topk_metric2dict
(metric_fn: Callable, topk_args: Sequence[int]) → Callable[source]¶ Logging wrapper for metrics with Sequence[Union[torch.Tensor, int, float, Dict]] output. Computes the metric and sync each element from the output sequence with passed topk argument.
 Parameters
metric_fn – metric function to compute
topk_args – topk args to sync outputs with
 Returns
wrapped metric function with List[Dict] output
 Raises
NotImplementedError – if metrics returned values are out of torch.Tensor, int, float, Dict union.

catalyst.metrics.functional.
wrap_class_metric2dict
(metric_fn: Callable, class_args: Sequence[str] = None) → Callable[source]¶ # noqa: D202 Logging wrapper for metrics with torch.Tensor output and [num_classes] shape. Computes the metric and sync each element from the output Tensor with passed class argument.
 Parameters
metric_fn – metric function to compute
class_args – class names for logging. default: None  class indexes will be used.
 Returns
wrapped metric function with List[Dict] output