Create yml configuration file with the following structure:
info: name: Name of a DAG project: Name of your project layout: Name of your layout. Please, consider layout section expdir: root folder of your project executors: # declaring DAG structure executor_a: type: executor_a # name of your executor executor_b: type: exectuor_b depends: executor_a executor_c: type: executor_c depends: [executor_a, executor_b] # if your node depends on several components
MLComp has already had some very useful executors. For example, Catalyst. It can be used to train your deep neural networks.
In case of a desire to create your own, inherit the executor’s class fom mlcomp.worker.executors.base.Executor.
Put it in any .py file, MLComp will use reflexion to find it.
# MLComp will import a module that contains the class with the specified name # (register does not matter). @Executor.register class Executor_A(Executor): def work(self): # do some useful work pass @classmethod def _from_config( cls, executor: dict, config: Config, additional_info: dict ): # initialize your executor with the params you specified in the configuration file # they are available in the executor variable return cls(...)
Some service fields in an executor configuration are the following:
gpu: 3 # you can specify requirements: gpu, cpu, memory(GB) # gpu can be set with a range; for example, 3-4 cpu: 1 memory: 0.1 distr: True # use distributed training single_node: True # run only on a single work computer depends: either string or list # create a structure of your DAG grid: list of configurations # more details on a gird_search page