Library Reorganization for v0.2.1 (v0.3.0) release

[iancze]

Massive WIP branch to collect several library improvement efforts. These will most likely involve a large and coordinated change to the structure of the codebase, so I'm grouping them together so that they land as a single PR rather than incremental and uncoordinated changes to main.

These changes are grouped thematically by their aim to improve the quality of MPoL as a PyTorch library

• improvements to stability of core routines
• aligning core usage patterns with established PyTorch idioms (e.g., prioritizing torch.tensor ahead of numpy.array , thinking about memory locations of arrays during optimization loops) should yield speed and stability improvements
• documentation changes to be up front about how MPoL (at least the core package) is designed to function as an interferometric imaging library for PyTorch, which might lead to further discussions about what is in/out of scope for the core 'plumbing' package, and what might make more sense in visread (pure numpy visibility manipulations and plotting) or another MPoL-dev package (as 'plumbing', to use Git's terminology).

Updates

• the proposed changes under consideration are now tracked by the "Architecture + Design" GitHub project board on MPoL-dev (available internally).
• Milestone was closed as redundent.

In rough order of planned approach:

Coverage, bug-fix, and 'foundational' changes

• Finish up type hinting started by Getting started with type hints #54, marking modules with 100% coverage with disallow_untyped_defs to prevent regressions.
  • Should also try to fix Inconsistent documentation for classes with .from_image_properties class method #233 mypy error with forward references.
  • Typing will likely reveal some inconsistent usage patterns for torch.tensor and np.array, which will beget further architectural redesign.
  • Identify routines that are under-tested and strengthen for the refactor.
• Fix loss function bugs and inconsistencies
  • log_likelihood loss incorrect and other loss names misleading #237
  • TV/TSV regularizer is not equivalent to source #153
  • Make loss functions and regularizers classes that inherit from torch.nn #131
  • Hermitian pairs of visibilities for imaging and fitting #100
• Remove mpol.utils.convert_baselines and mpol.utils.broadcast_and_convert_baselines, since this functionality now exists in visread (mpol.utils.convert_baselines and mpol.utils.broadcast_and_convert_baselines duplicate visread functionality #227).
• Rather than pre-package a mock ALMA logo dataset on Zenodo, create this in-package, on-the-fly, using NuFFT and some reference baselines.
• Change base unit from klambda to lambda
  • Base unit for spatial frequency #223
  • also simultaneous changes to visread

Changes to introduce Stochastic Gradient Descent workflow

• Add DdidSampler to codebase. Create a documentation page showing how to create a TensorDataset and a DataLoader Redesign UVDataset with Pytorch idioms in mind #162
• Documentation/"Getting Started" to explain how MPoL library usage follows the the normal ML paradigm with SGD (Framing model.forward more like ML libraries? #188)

Further documentation changes

• Reorganize SimpleNet and the entire GriddedDataset workflow as an alternative option to SGD, with documentation about the use cases when one or the other might be preferred.
• Consolidate or remove redundant tutorials.
• Think about what content makes sense as rendered docs (code demonstrating key library functionality, building blocks, preferably concise) or as a longer .py file in examples/ (actual workflows following official pytorch/examples. E.g.,
  • reduce scope of ALMA Logo tutorial upgrades #75 to a "Quickstart/Getting Started" with links to documentation further explaining functionality where necessary
  • Direct longer content to examples or a new MPoL-dev package implementing 'plumbing' or tutorials.
    • Update docs to demo running a fit with pipeline #134
    • HD 143006 imaging tutorial overview #25
    • Start examples directly from DSHARP datasets, removing need to download processed datasets from Zenodo & demonstrates required CASA work to user
• Make sure all routines render in the API docs
• Detailed migration/updated instructions from v0.2.0 => v0.2.1 in changelog, e.g. in the style of Julia Release Notes.

Though, if we manage to accomplish everything I've listed here a v0.3.0 makes more sense than a v0.2.1.

[iancze]

Superseded by #243 after branch name v0.2.1 => v0.3.0