[ENH] `torch` / `lightning` integration

[fkiraly]

It would be nice to have integration with basic torch and lightning tuning workflows, to allow autoML style tuning.

This would require a BaseExperiment descendant class TorchExperiment which takes a DataLoader, a LightningModule, and a Trainer, performs a training run, and returns the validation loss/score. Also see the extension template https://github.com/SimonBlanke/Hyperactive/blob/main/extension_templates/experiments.py

Since tuning is not API preserving in torch / lightning, I would suggest an additional function that produces parameters for a tuned network, or the initialized tuned network with said parameters - e.g., tune_lightning(optimizer: BaseOptimizer, loader, module, trainer)? Or is it clear enough to construct the optimizer and call solve?

Comments appreciated on the design.

[fkiraly]

If it works in this generality, it should be tested whether it can be used to replace or extend the pytorch-forecasting tuning feature - which is currently optuna based.

FYI @phoeenniixx, @PranavBhatP, @agobbifbk

[amitsubhashchejara]

Hello @fkiraly, I would like to take on this issue. Can you elaborate more on this and how we can get started?

[fkiraly]

Sure! The first step would be to write the TorchExperiment, based on BaseExperiment, as described above. The _evaluate function should be implemented, and internally it just runs the experiment as described. It gets the trainer, dataloader, etc, passed in __init__ and runs the entire workflow until you get the validation loss.

Let us know here or on discord if you need more detailed pointers!

[amitsubhashchejara]

Sure, @fkiraly, please provide me with detailed pointers to get started! This will give a better start.

[amitsubhashchejara]

I understand that a new class, Torch Experiment, should be implemented to return the validation loss. Also, a new function should be implemented to preserve and return the parameters of the tuned network. Is my understanding right?
Also, when you say “produce parameters for a tuned network,” you want something like: I supply a base model definition, run “tuning,” and then get back (a) a new model instance with the tuned parameters, or (b) just the tuned parameter values (a state_dict or similar) that can be loaded into a model with the same architecture.
I would greatly appreciate detailed pointers!

[fkiraly]

I would suggest, implement the torch experiment first, the rest can build on that later.

[amitsubhashchejara]

Sure, I'll start!

[amitsubhashchejara]

Hello @fkiraly, I have a question. To perform a training run and return a validation loss/score, we need two dataloaders, a train_loader and a validation_loader. So should the TorchExperiment class accept two dataloaders? Also, which score should be returned along with the loss? Accuracy, AUCROC, F1?

[amitsubhashchejara]

Maybe the class can accept a datamodule, or both train_loader and cv_loader should be provided. For the score, we can set val_loss as the default.

[fkiraly]

Yes, I was thinking of using a lightning DataModule, which bundles both training and validation loader.

I would advise, let´s start with lightning based parametrization, we can later think about whether we would also like a torch native one.

Also, which score should be returned along with the loss? Accuracy, AUCROC, F1?

I think that should be up to the user.

[amitsubhashchejara]

Okay thankyou! There is one more point: should the trainer run on the original LightningModule or on a copy of the same? Because we want to save the params of the model, the user has to use the save_hyperparameters in the __init__ of the LightningModule, only then can we create a new instance in the _paramnames function with the same params.