Shortcuts

Dataflow

Base dataflow

Catalyst uses the “key-value is all you need” approach. In other words, the output of your Dataset/Dataloader should be a key-value (python dict).

Example of dataflow:

class MyDataset:

    def __get_item__(self, index):
        ...
        return {"features": np.ndarray, "targets": np.ndarray}

class MyModel:

    def forward(self, features):
        ...
        return logits

class MyRunner:

    def handle_batch(self, batch):
        # on this step we also have self.input = batch = {"features": ..., "targets": ...}
        logits = self.model(batch["features"])
        loss = self.criterion(logits, batch["targets"])
        self.output = {"logits": logits}
        # this is useful for other components of the pipiline

loader = MyDataset()
model = MyModel()
runner = MyRunner()
runner.train(...)

Note

SupervisedRunner has data preprocessing features to transform tuple/list-based data into key-value.

Such approach is easily extensible for any number of features, targets and very convenient to read, thanks to “automatic naming documentation” - keys for the values:

class MyDataset:

    def __get_item__(self, index):
        ...
        return {"features": np.ndarray, "extra_features": np.ndarray, "targets": np.ndarray}

class MyModel:

    def forward(self, features, extra_features):
        ...
        return logits

class MyRunner:

    def handle_batch(self, batch):
        # on this step we also have self.input = batch = {"features": ..., "extra_features": ...,"targets": ...}
        logits = self.model(batch["features"], batch["extra_features"])
        loss = self.criterion(logits, batch["targets"])
        self.output = {"logits": logits}
        # this is useful for other components of the pipiline

loader = MyDataset()
model = MyModel()
runner = MyRunner()
runner.train(...)

Moreover, if some of the features are not required anymore - you don’t have to rewrite your code:

class MyDataset:

    def __get_item__(self, index):
        ...
        return {"features": np.ndarray, "extra_features": np.ndarray, "targets": np.ndarray}

class MyModel:

    def forward(self, features):
        ...
        return logits

class MyRunner:

    def handle_batch(self, batch):
        # on this step we also have self.input = batch = {"features": ..., "extra_features": ...,"targets": ...}
        logits = self.model(batch["features"])
        loss = self.criterion(logits, batch["targets"])
        self.output = {"logits": logits}
        # this is useful for other components of the pipiline

loader = MyDataset()
model = MyModel()
runner = MyRunner()
runner.train(...)

Key-value storage also can be used to store the datasets/loaders for the experiment. In this case we also need to use OrderedDict to ensure correct epoch handling - that your model will firstly train on some train dataset and only then will be evaluated on some valid dataset:

train_loader = MyDataset(...)
valid_loader = MyDataset(...)
loaders = OrderedDict("train": train_loader, "valid": valid_loader)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders)

Catalyst uses the following “automatic naming documentation” for loader keys handling:

  • if loader_key starts with “train” - is’s our train datasoure, we need to run forward and backward passes on it.

  • if loader_key starts with “valid” - is’s our validation datasoure, we need to run forward, but not the backward pass on it.

  • if loader_key starts with “infer” - is’s our datasoure for model inference, we need to run forward, but not the backward pass on it.

Multiple datasources

Thanks to key-value approach, it’s possible to handle any number of datasets/loader without code changes or tricks with Datasets concatination, etc:

train_loader = MyDataset(...)
train2_loader = MyDataset(...)
valid_loader = MyDataset(...)
valid2_loader = MyDataset(...)
loaders = OrderedDict(
    "train": train_loader,
    "train2": train2_loader,
    "valid": valid_loader,
    "valid2": valid2_loader,
)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders)

What is even more interesting, you could also do something like:

train_loader = MyDataset(...)
train2_loader = MyDataset(...)
valid_loader = MyDataset(...)
valid2_loader = MyDataset(...)
loaders = OrderedDict(
    "train": train_loader,
    "valid": valid_loader,
    "train2": train2_loader,
    "valid2": valid2_loader,

)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders)

Once again, it’s also valid to do something like:

train_loader = MyDataset(...)
train2_loader = MyDataset(...)
valid_loader = MyDataset(...)
valid2_loader = MyDataset(...)
loaders = OrderedDict(
    "train": concat_datasets(train_loader, train2_loader),
    "valid": concat_datasets(valid_loader, valid2_loader),
)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders)

Loader for model selection

In case of multiple loaders, you could select one of them for model selection with valid_loader param in the runner.train. For example, to use valid2 loaders as your model selection one you could do the following:

train_loader = MyDataset(...)
train2_loader = MyDataset(...)
valid_loader = MyDataset(...)
valid2_loader = MyDataset(...)
loaders = OrderedDict(
    "train": train_loader,
    "train2": train2_loader,
    "valid": valid_loader,
    "valid2": valid2_loader,
)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders, valid_loader="valid2")

Note

By default, Catalyst suppose to use valid_loader=valid for model selection.

Metric for model selection

Suppose, you are using set of different metrics in your pipeline:

class MyRunner:

    def handle_batch(self, batch):
        # on this step we also have self.input = batch = {"features": ..., "targets": ...}
        logits = self.model(batch["features"])
        loss = self.criterion(logits, batch["targets"])
        accuracy01, accuracy03 = accuracy(logits, batch["targets"], topk=(1, 3))
        self.batch_metrics.update(**{
            "loss": loss,
            "accuracy01": accuracy01,
            "accuracy03": accuracy03,
        })
        self.output = {"logits": logits}
        # this is useful for other components of the pipiline

loaders = ...
model = ...
runner = MyRunner()
runner.train(model=model, loaders=loaders)

You could select one for model selection with main_metric and minimize_metric params in the runner.train. For example, to use accuracy01 metric as your model selection one you could do the following:

class MyRunner:

    def handle_batch(self, batch):
        # on this step we also have self.input = batch = {"features": ..., "targets": ...}
        logits = self.model(batch["features"])
        loss = self.criterion(logits, batch["targets"])
        accuracy01, accuracy03 = accuracy(logits, batch["targets"], topk=(1, 3))
        self.batch_metrics.update(**{
            "loss": loss,
            "accuracy01": accuracy01,
            "accuracy03": accuracy03,
        })
        self.output = {"logits": logits}
        # this is useful for other components of the pipiline

loaders = ...
model = ...
runner = MyRunner()
# as far as we would like to maximize our model accuracy...
runner.train(model=model, loaders=loaders, main_metric="accuracy01", minimize_metric=False)

Note

By default, Catalyst suppose to use main_metric=loss and minimize_metric=False for model selection.

Use part of the data

If you would like to use only some part of your data from the loader (for example, you would like to overfit for one small portion of the data to check your pipeline), you could use BatchLimitLoaderWrapper:

train_loader = BatchLimitLoaderWrapper(MyDataset(...), num_batches=1)
valid_loader = MyDataset(...)
loaders = OrderedDict("train": train_loader, "valid": valid_loader)
model = MyModel()
runner = MyRunner()
runner.train(model=model, loaders=loaders)

As a more user-friendly approach with runner.train:

train_loader = MyDataset(...)
valid_loader = MyDataset(...)
loaders = OrderedDict("train": train_loader, "valid": valid_loader)
model = MyModel()
runner = MyRunner()
# here we overfit for one batch per loader
runner.train(model=model, loaders=loaders, overfit=True)

And more convenient and customizable way:

train_loader = MyDataset(...)
valid_loader = MyDataset(...)
loaders = OrderedDict("train": train_loader, "valid": valid_loader)
model = MyModel()
runner = MyRunner()
# here we overfit for 10 batches in `train` loader
# and half of the `valid` loader
runner.train(
    model=model,
    loaders=loaders,
    callbacks=[dl.BatchOverfitCallback(train=10, valid=0.5)]
)

If you haven’t found the answer for your question, feel free to join our slack for the discussion.