Skip to content

Package installs update #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
grallewellyn merged 3 commits into from
Oct 5, 2023
Merged

Package installs update #341

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2d5fd06
converted some mambas to conda installs, working on custom environments
grallewellyn Sep 27, 2023
b856c92
changed all mentions of mamba to conda
grallewellyn Sep 27, 2023
2bd647b
fixed respository bug
grallewellyn Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/source/_static/launch_custom_kernel_conda.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/source/_static/launch_registered_kernel_custom_conda.png
View file
Open in desktop
Binary file not shown.
8 changes: 4 additions & 4 deletions docs/source/getting_started/writing_code.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Clone a Respository with GitHub\n",
"### Clone a Repository with GitHub\n",
"\n",
"Any git repository online can be used to host code. Here is an example algorithm you can use for this getting started guide:\n",
"https://github.com/MAAP-Project/dps-unit-test\n",
Expand Down Expand Up @@ -84,7 +84,7 @@
"1. Make a new folder for your test algorithm. Open a terminal here (File > New > Terminal or use the blue \"+\" button above the Jupyter file browser).\n",
"![New Folder](_static/clone_demo1.png)\n",
"\n",
"2. Follow the steps [above](#Clone-a-Respository-with-GitHub) for how to Clone a Respository with GitHub"
"2. Follow the steps [above](#Clone-a-Repository-with-GitHub) for how to Clone a Repository with GitHub"
]
},
{
Expand Down Expand Up @@ -225,9 +225,9 @@
"source": [
"## Customizing your workspace environment\n",
"\n",
"Your Jupyter workspace has a set of pre-installed libraries, depending on [which Stack you selected](getting_started.ipynb#Creating-a-workspace). If you need libraries that are not pre-installed, we suggest using an environment manager; `mamba` is pre-installed to help with this. \n",
"Your Jupyter workspace has a set of pre-installed libraries, depending on [which Stack you selected](getting_started.ipynb#Creating-a-workspace). If you need libraries that are not pre-installed, we suggest using an environment manager; `conda` is pre-installed to help with this. \n",
"\n",
"Full [documentation on configuring mamba](../system_reference_guide/custom-environments.ipynb) (or conda) may be found in the [System Reference Guide](../system_reference.rst).\n"
"Full [documentation on configuring conda](../system_reference_guide/custom-environments.ipynb) may be found in the [System Reference Guide](../system_reference.rst).\n"
]
},
{
Expand Down
45 changes: 26 additions & 19 deletions docs/source/system_reference_guide/custom-environments.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,27 +38,33 @@
"\n",
Copy link
Collaborator

@wildintellect wildintellect Sep 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bold or italicize libmamba

use a callout box to talk about libmamba

> Note:
text

Reply via ReviewNB

Copy link
Member Author

@grallewellyn grallewellyn Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wildintellect I agree with bold or italicize libmamba, but how would you suggest I change the line:
"We use conda with the libmamba solver as a package manager to install, update or remove packages (libraries)."? libmamba is in a callout box even when it is in the middle of the sentence?

Copy link
Collaborator

@wildintellect wildintellect Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the callout box should be after the sentence explaining what libmamba is and how you can use classic if you want/need

Copy link
Member Author

@grallewellyn grallewellyn Sep 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So conda install --solver=classic -c conda-forge xarray is in the callout box?

Copy link
Member Author

@grallewellyn grallewellyn Oct 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wildintellect Just following up on the above comment

"### Package manager\n",
"\n",
"We use `mamba` (a fast `conda` drop-in replacement) as a package manager to install, update or remove packages (libraries). `mamba` works with 'environments' that are directories in your local file system containing a set of packages. When you work 'in a given environment', it means that your programs will look for dependencies in that environment's `mamba` directory. All workspaces launch with an environment called `base`, which is a `mamba` environment that has all the pre-installed libraries. If you open a terminal launcher after creating a `Basic Stable` workspace : \n",
"We use `conda` with the libmamba solver as a package manager to install, update or remove packages (libraries). `conda` works with 'environments' that are directories in your local file system containing a set of packages. When you work 'in a given environment', it means that your programs will look for dependencies in that environment's `conda` directory. All workspaces launch with an environment called `base`, which is a `conda` environment that has all the pre-installed libraries. If you open a terminal launcher after creating a `Basic Stable` workspace : \n",
"\n",
"\n",
"![Base conda environment location](../_static/base_conda_environment_location.png)\n",
"\n",
"You can notice that a `base` `mamba` environment is activated, and its libraries are located in `/opt/conda`. \n",
"You can notice that a `base` `conda` environment is activated, and its libraries are located in `/opt/conda`. \n",
"\n",
"### Extending the `base` environment in a given workspace session.\n",
"\n",
"**Note : any modification to the `base` environment does not survive a workspace restart. In other words, modifications to `/opt/conda` disappear after a workspace restart**. \n",
"\n",
"Extending an existing `mamba` environment means adding packages on top of what it contains, which works provided there are no dependency conflicts. You can install libraries using the `mamba install` command to install additional packages in your current environment (run `mamba --help` to learn more about how to use `mamba` commands). For example :\n",
"Extending an existing `conda` environment means adding packages on top of what it contains, which works provided there are no dependency conflicts. You can install libraries using the `conda install` command to install additional packages in your current environment (run `conda --help` to learn more about how to use `conda` commands). All `conda` install commands should use `-c conda-forge` otherwise it's unlikely to work, since many/most of the packages installed already are from conda-forge. For example :\n",
"\n",
"```\n",
"mamba install xarray\n",
"conda install -c conda-forge xarray\n",
"```\n",
"\n",
"libmamba is the default solver, but users are welcome to set the solver to \"classic\" with: \n",
"\n",
"```\n",
"conda install --solver=classic -c conda-forge xarray\n",
"```\n",
"\n",
"However, it is recommended to use configuration files for reproducibility and shareability. With this approach, assuming your configuration file is named `config.yml`, the command to use is : \n",
"\n",
"```\n",
"mamba env update -f config.yml\n",
"conda env update -f config.yml\n",
"```\n",
"\n",
"For more details on configuration files, see the [Custom environments section](#Custom-environments) and for an example of this command, refer to the [subsection about updating an environment with a configuration file](#Updating-an-existing-environment-with-a-configuration-file).\n"
Expand All @@ -73,7 +79,7 @@
"\n",
"*For the rest of this README, in each section we provide a link to download an example YAML configuration file.*\n",
"\n",
"You can use the `mamba` CLI to create a new, custom environment. The parameters (the list of libraries, the location where to search for them, etc...) can be passed either from a configuration YAML file or directly on the console. We recommend using the first option (a YAML file is easier to share and modify). \n",
"You can use the `conda` CLI to create a new, custom environment. The parameters (the list of libraries, the location where to search for them, etc...) can be passed either from a configuration YAML file or directly on the console. We recommend using the first option (a YAML file is easier to share and modify). \n",
"\n",
"### Basic custom environment\n",
"\n",
Expand All @@ -82,39 +88,39 @@
"This configuration installs specific versions `python`, `pandas` and `geopandas` from `conda-forge`. If versions aren't specified, the latest is installed. We recommend to always specify the version for reproducibility. The basic command to create this environment would be :\n",
"\n",
"```\n",
"mamba env create -f env-example.yml\n",
"conda env create -f env-example.yml\n",
"```\n",
"\n",
"However, this stores this environment files in `/opt/conda`, which is a directory that is recreated when the workspace restarts, and so custom environments are lost. Therefore, you want to specify a storage location in your user directory with the `--prefix` parameter\n",
"\n",
"```\n",
"mamba env create -f env-example.yml --prefix /projects/env\n",
"conda env create -f env-example.yml --prefix /projects/env/env-example\n",
"```\n",
"\n",
"and to activate it : \n",
"\n",
"```\n",
"mamba activate env-example\n",
"conda activate /projects/env/env-example\n",
"```\n",
"\n",
"### Updating an existing environment with a configuration file\n",
"\n",
"*Example config file for this section [here](./example_conda_configuration_files/env-extend.yml).*\n",
"\n",
"You can *update* an existing environment with a configuration file as well. For example, let's assume you have a `mamba` environment with a set of packages already installed in it (for example the `base` environment), but it doesn't have `xarray` and `geopandas`. Using the linked example config : \n",
"You can *update* an existing environment with a configuration file as well. For example, let's assume you have a `conda` environment with a set of packages already installed in it (for example the `base` environment), but it doesn't have `xarray` and `geopandas`. Using the linked example config : \n",
"\n",
"```\n",
"mamba env update -f env-extend.yml\n",
"conda env update -f env-extend.yml\n",
"```\n",
"\n",
"This command will update `base` by adding `xarray` and `geopandas`, provided it does not cause conflicts with the existing libraries. \n",
"This command will update the active environment by adding `xarray` and `geopandas`, provided it does not cause conflicts with the existing libraries. \n",
"\n",
"\n",
"### Using `pip` for python packages\n",
"\n",
"*Example config file for this section [here](./example_conda_configuration_files/env-with-pip.yml).*\n",
"\n",
"Some python packages might not be availabe in the channel you are using, or in any `mamba` channel. If that package however is in `PyPI` (the official python package repository), one can use `pip` within a `mamba` environment to download packages. The recommended way is to specify this in the configuration file. In the linked example, we add `stackstac` as a dependency to install from `PyPI` because it is not available in the `conda-forge` channel. \n",
"Some python packages might not be availabe in the channel you are using, or in any `conda` channel. If that package however is in `PyPI` (the official python package repository), one can use `pip` within a `conda` environment to download packages. The recommended way is to specify this in the configuration file. In the linked example, we add `stackstac` as a dependency to install from `PyPI` because it is not available in the `conda-forge` channel. \n",
"\n",
"### Using custom environments in jupyter notebooks\n",
"\n",
Expand All @@ -126,23 +132,24 @@
"- Create your environment using the linked configuration file.\n",
"- Install the environment as a kernel by running the following command (parameter values follow the example mentioned):\n",
" ```\n",
" python -m ipykernel install --user --name env-with-ipykernel --display-name \"Python env-with-ipykernel\".\n",
" python -m ipykernel install --user --name env-with-ipykernel --display-name \"Python env-with-ipykernel\"\n",
" ```\n",
" The above command installs the environment as a kernel in Jupyter, making it accessible in the notebook with a display name of \"Python env-with-ipykernel\".\n",
"- Wait around 30 seconds and launch a new notebook. Among the kernel options, you should see \"Python env-with-ipykernel\" listed. Below you can see a screenshot that shows what this step looks like:\n",
"![Register a kernel with a conda environment and launch a notebook with it](../_static/launch_registered_kernel_custom_conda.png)\n",
"![Register a kernel with a conda environment and launch a notebook with it](../_static/launch_custom_kernel_conda.png)\n",
"- Remove by listing kernelspecs `jupyter kernelspec list` to find name, then `jupyter kernelspec remove <env>`\n",
"\n",
"### Suggested packages for custom environment\n",
"\n",
"*Example config file for this section [here](./example_conda_configuration_files/env-with-maap-py.yml)*\n",
"\n",
"MAAP users typically use the python `maap-py`. It's pre-installed in all workspaces, in the `base` mamba environment, but any custom environment should specify it, otherwise it is not going to be accessible from that environment. However, `maap-py` is not packaged in a public package repository, like `PyPI` or `conda-forge`. It is possible to install it directly from its github repository with `pip` though. See the configuration example linked. You can note that in the example, `maap-py` is 'versioned' using a commit hash (at the end of the github URL). "
"MAAP users typically use the python `maap-py`. It's pre-installed in all workspaces, in the `base` conda environment, but any custom environment should specify it, otherwise it is not going to be accessible from that environment. However, `maap-py` is not packaged in a public package repository, like `PyPI` or `conda-forge`. It is possible to install it directly from its github repository with `pip` though. See the configuration example linked. You can note that in the example, `maap-py` is 'versioned' using a commit hash (at the end of the github URL). "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3.11.1 64-bit",
"language": "python",
"name": "python3"
},
Expand All @@ -156,7 +163,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.10"
"version": "3.11.1"
},
"metadata": {
"interpreter": {
Expand All @@ -165,7 +172,7 @@
},
"vscode": {
"interpreter": {
"hash": "8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1"
"hash": "5c7b89af1651d0b8571dde13640ecdccf7d5a6204171d6ab33e7c296e100e08a"
}
}
},
Expand Down
2 changes: 1 addition & 1 deletion docs/source/system_reference_guide/work_with_git.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"metadata": {},
"source": [
"### Committing Changes to a Git Repository\n",
"All projects imported from GitLab can be found in `/projects`. Instructions how to clone a repository from Git can be found [here](../getting_started/writing_code.html#Clone-a-Respository-with-GitHub) and instructions for working with MAAP GitLab can be found [here.](../getting_started/writing_code.html#Working-with-the-MAAP-GitLab)\n",
"All projects imported from GitLab can be found in `/projects`. Instructions how to clone a repository from Git can be found [here](../getting_started/writing_code.html#Clone-a-Repository-with-GitHub) and instructions for working with MAAP GitLab can be found [here.](../getting_started/writing_code.html#Working-with-the-MAAP-GitLab)\n",
"\n",
"Before you update your project with your changes, you need to get a classic personal access token from GitHub. MAAP currently doesn't supported fine-grained tokens. Create a classic personal access token by following [these steps](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) in Developer settings. Remember to save your personal access token for later use. \n",
"\n",
Expand Down