Skip to content

GEOPY-863: use new dev script to create lock files #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
domfournier merged 1 commit into from
May 27, 2024
Merged

GEOPY-863: use new dev script to create lock files #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 137 additions & 87 deletions README-dev.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,158 +7,211 @@ Setup for development
:depth: 2

After you have cloned the Git repository, you will need to:
#. create the ``.lock`` files for the dependencies
#. create a virtual ``conda`` environment for development, where to install the dependencies
of the project
#. execute the tests
#. setup Git LFS if needed
#. configure the pre-commit hooks for static code analysis and auto-formatting
#. configure the Python IDE (PyCharm)

Create the .lock files
^^^^^^^^^^^^^^^^^^^^^^
#. create the Conda environment lock files for the dependencies
#. create a virtual Conda environment for development, where to install the
dependencies of the project
#. execute the tests
#. setup Git LFS if needed
#. configure the pre-commit hooks for static code analysis and auto-formatting
#. configure the Python IDE (PyCharm)

First, you need to create the ``.lock`` files for the dependencies defined in ``pyproject.toml``.
To do so, double click on ``devtools\run-conda-lock.bat`` or execute it from command line::

$ [path\to\simpeg-drivers]\devtools\run_conda_lock.bat
.. _conda-lock: https://conda.github.io/conda-lock/
.. _Poetry: https://python-poetry.org/
.. _pipx: https://pipxproject.github.io/pipx/
.. _pre-commit: https://pre-commit.com/
.. _py-deps-lock: https://github.com/MiraGeoscience/py-deps-lock


Create the Conda lock files and environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Install internal tool ``py-deps-lock`` to create the Conda environment lock files,
and the local development environment. ``py-deps-lock`` is a wrapper around `conda-lock`_.

See details in the `py-deps-lock`_ documentation.

To install ``py-deps-lock`` with `pipx`_, run the command::

It will create ``.lock`` in the ``environments`` folder.
The created ``.lock`` files are the combination of python version and platforms.
The platforms are specified in ``conda-lock`` section of the ``pyproject.toml`` file:
$ pipx install git+https://github.com/MiraGeoscience/py-deps-lock

.. code-block:: toml

[tool.conda-lock]
platforms = ['win-64', 'linux-64']
This expose commands to the execution ``PATH``:

The python versions are specified at the beginning of the ``devtools\run-conda-lock.py`` file:
- create-dev-env
- deps-lock

.. code-block:: python

_python_versions = ["3.10"]
Create the Conda environment lock files
---------------------------------------
First, create the Conda environment lock files (``*.conda.lock.yml``) for the dependencies defined
in `pyproject.toml`_. From the root of the project, run the command::

The ``Install_or_Update.bat`` and the ``setup-dev.bat`` will use them to install the environment.
$ deps-lock

Anytime dependencies are added or removed to the ``pyproject.toml`` file, you need to run ``run-conda-lock.bat`` again. Run it as well when you want to fetch newly available versions of the dependencies (typically patches, still in accordance with the specifications expressed in ``pyproject.toml``).
It will create or replace the ``*.conda.lock.yml`` files in the ``environments`` folder:
one for runtime dependencies, and one for development dependencies (with the ``-dev`` suffix),
for each combinations of Python versions and platforms
(platforms are specified in ``conda-lock`` section of the ``pyproject.toml``).

Install the conda environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
These files will be used by installation script and to create the development environment.

For development, you need a **conda** environments. you can install it running the ``setup-dev.bat`` or::

$ [path\to\simpeg-drivers]\devtools\setup-dev.bat
Install the Conda environment
-----------------------------

For development, you need a **Conda** environment. From the root of the project, run the command::

$ create-dev-env

This command install a local environment at the base of your repository: ``.conda-env``.
This environment should automatically be recognized by the conda installation.
This environment should automatically be recognized by the Conda installation.

Configure the pre-commit hooks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To activate this environment, run the following command from the root of the project::

`pre-commit`_ is used to automatically run static code analysis upon commit.
The list of tools to execute upon commit is configured in the file `.pre-commit-config.yaml`_.
$ conda activate ./.conda-env

pre-commit can be installed using a Python installation on the system, or one from a conda environment.

- To install pre-commit using Python (and pip) in your system path:
Updating dependencies
^^^^^^^^^^^^^^^^^^^^^

.. code-block:: bash
Dependencies are listed in `pyproject.toml`_ with version constraints.
Versions are then locked using ``deps-lock`` as previously described.

pip install --user pre-commit
Anytime dependencies are added to or removed from the ``pyproject.toml`` file,
regenerate the Conda environment lock files, re-running ``deps-lock`` from command line.

- Or to install from an activated conda environment:
Regenerate the Conda environment lock files as well when you want to fetch newly
available versions of the dependencies (typically patches, still in accordance with
the specifications expressed in ``pyproject.toml``).

.. code-block:: bash

conda install -c conda-forge pre-commit
Adding a dependency
-------------------
First install the dependency using ``conda``:

Then, in either way, install the pre-commit hooks as follow (**current directory is the project folder**):
$ conda install -p ./.conda-env my_new_dep

.. code-block:: bash
Then update the list of dependencies in `pyproject.toml`_ with a suited version constraint
(if for development only, place it under section ``[tool.poetry.group.dev.dependencies]``).

pre-commit install
For example, if ``conda`` installed version 1.5.2 of ``my_new_dep``,
then add ``my_new_dep="^1.5.2"``.

To prepare and check the commit messages, you can also use the following commands:
Do not forget to regenerate the Conda environment lock files.

.. code-block:: bash

pre-commit install -t prepare-commit-msg -t commit-msg
How to use **Poetry** to update the dependency list (optional)
--------------------------------------------------------------
`Poetry`_ provides a command line interface to easily add or remove dependencies:

It configures ``pre-commit`` to prepares and checks the commit ensuring it has a JIRA issue ID: if no ID was provided, it extracts it from the branch name. If one was provided, it checks it is the same one as in the branch name.
(path/to/.conda-env) $ poetry add another_package --lock

To run pre-commit manually, use the following command:
Note the ``--lock`` option, that simple creates or updates the lock file, without Poetry installing anything.
``poetry`` would install the package through ``pip`` while we want dependencies to be installed through ``conda``
so that they match the version pinned by ``conda-lock``.

.. code-block:: bash
One limitation though: Poetry will look for packages in PiPY only and not in the Conda channels.
The version selected by Poetry might thus not be available for Conda.

pre-commit run --all-files
To install ``Poetry`` on your computer, refer to the `Poetry`_ documentation.

To run only on changes staged for commit:

.. code-block:: bash
Configure the pre-commit hooks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

pre-commit run
`pre-commit`_ is used to automatically run static code analysis upon commit.
The list of tools to execute upon commit is configured in the file `.pre-commit-config.yaml`_.

If a tool fails running, it might be caused by an obsolete versions of the tools that pre-commit is trying to execute.
Try the following command to update them:
pre-commit can be installed using a Python installation on the system, or one from a Conda environment,
or through `pipx`_.

.. code-block:: bash
- To install ``pre-commit`` using ``pipx`` (recommended)::

pre-commit autoupdate
$ pipx install pre-commit

Upon every commit, all the pre-commit checks run automatically for you, and reformat files when required. Enjoy...
- To install ``pre-commit`` using Python (and pip) in your system path::

If you prefer to run pre-commit upon push, and not upon every commit, use the following commands:
$ pip install --user pre-commit

.. code-block:: bash
- Or to install from an activated Conda environment::

pre-commit uninstall -t pre-commit
pre-commit install -t pre-push
$ conda install -c conda-forge pre-commit

Then, in either way, install the pre-commit hooks as follow (**current directory is the project folder**)::

$ pre-commit install

To prepare and check the commit messages, you can also use the following commands::

$ pre-commit install -t prepare-commit-msg -t commit-msg

It configures ``pre-commit`` to prepares and checks the commit ensuring it has a JIRA issue ID:
if no ID was provided, it extracts it from the branch name;
if one was provided, it checks it is the same one as in the branch name.

To run pre-commit manually, use the following command::

$ pre-commit run --all-files

To run only on changes staged for commit::

$ pre-commit run

If a tool fails running, it might be caused by an obsolete versions of the tools that pre-commit is
trying to execute. Try the following command to update them::

$ pre-commit autoupdate

Upon every commit, all the pre-commit checks run automatically for you, and reformat files when required. Enjoy...

If you prefer to run pre-commit upon push, and not upon every commit, use the following commands::

$ pre-commit uninstall -t pre-commit
$ pre-commit install -t pre-push

.. _pre-commit: https://pre-commit.com/

Running the tests
^^^^^^^^^^^^^^^^^

Test files are placed under the ``tests`` folder. Inside this folder and sub-folders,
Python test files are to be named with ``_test.py`` as a suffix.
The test function within this files must have a ``test_`` prefix.


Install pytest
--------------

.. _pytest: https://docs.pytest.org/

If you installed your environment through ``setup-dev.bat``, pytest is already installed.
You can run it from the conda command (**in your project folder**):
You can run it from the Conda command (**in your project folder**)::

.. code-block:: bash
$ pytest tests

pytest tests

Code coverage with Pytest
-------------------------
.. _pytest-cov: https://pypi.org/project/pytest-cov/

If you installed your environment through ``setup-dev.bat``, `pytest-cov`_ is already installed.
It allows you to visualize the code coverage of your tests.
You can run the tests from the console with coverage:

.. code-block:: bash
You can run the tests from the console with coverage::

pytest --cov=simpeg_drivers --cov-report html tests
$ pytest --cov-report html tests

The html report is generated in the folder ``htmlcov`` at the root of the project.
You can then explore the report by opening ``index.html`` in a browser.


Git LFS
^^^^^^^
In the case your package requires large files, `git-lfs`_ can be used to store those files.
Copy it from the `git-lfs`_ website, and install it.

Then, in the project folder, run the following command to install git-lfs:

.. code-block:: bash
Then, in the project folder, run the following command to install git-lfs::

git lfs install
$ git lfs install


It will update the file ``.gitattributes`` with the list of files to track.
Expand All @@ -167,26 +220,26 @@ Then, add the files and the ``.gitattributes`` to the git repository, and commit

.. _git-lfs: https://git-lfs.com/

Then, add the files to track with git-lfs:
Then, add the files to track with git-lfs::

.. code-block:: bash
$ git lfs track "*.desire_extension"

git lfs track "*.desire_extension"

IDE : PyCharm
^^^^^^^^^^^^^
`PyCharm`_, by JetBrains, is a very good IDE for developing with Python.


Configure the Python interpreter in PyCharm
--------------------------------------------

First, excluded the ``.conda-env`` folder from PyCharm.
Do so, in PyCharm, right-click on the ``.conda-env`` folder, and ``Mark Directory as > Excluded``.

Then, you can add the conda environment as a Python interpreter in PyCharm.
Then, you can add the Conda environment as a Python interpreter in PyCharm.

.. image:: devtools/images/pycharm-exclude_conda_env.png
:alt: PyCharm: Exclude conda environment
:alt: PyCharm: Exclude Conda environment
:align: center
:width: 40%

Expand All @@ -203,12 +256,12 @@ Select ``Conda Environment``, ``Use existing environment``,
and select the desired environment from the list (the one in the ``.conda-env`` folder):

.. image:: devtools/images/pycharm-set_conda_env_as_interpreter.png
:alt: PyCharm: Set conda environment as interpreter
:alt: PyCharm: Set Conda environment as interpreter
:align: center
:width: 80%

Then you can check the list of installed packages in the ``Packages`` table. You should see
**simpeg-drivers** and its dependencies. Make sure to turn off the ``Use Conda Package Manager``
this source package and its dependencies. Make sure to turn off the ``Use Conda Package Manager``
option to see also the packages installed through pip:

.. image:: devtools/images/pycharm-list_all_conda_packages.png
Expand Down Expand Up @@ -236,6 +289,7 @@ PyCharm will nicely present the test results and logs:
:align: center
:width: 80%


Execute tests with coverage from PyCharm
----------------------------------------

Expand All @@ -256,10 +310,11 @@ then add the following option in the ``Additional Arguments`` field:

select ``pytest in tests``, and add the following option in the ``Additional Arguments`` field::

--cov=simpeg_drivers --cov-report html
--cov-report html

Then, run the tests as usual, and you will get a nice report of the code coverage.


Some useful plugins for PyCharm
--------------------------------
Here is a suggestion for some plugins you can install in PyCharm.
Expand All @@ -278,11 +333,6 @@ Here is a suggestion for some plugins you can install in PyCharm.
.. _.pre-commit-config.yaml: .pre-commit-config.yaml


License
^^^^^^^
# TODO: ADD LICENSE TERMS


Copyright
^^^^^^^^^
Copyright (c) 2024 Mira Geoscience Ltd.
Loading