From 3c01edfd2e622d2797429ca424c810e83ab7d85e Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Thu, 4 Sep 2025 14:22:31 -0700 Subject: [PATCH 01/11] start draft of uv docs --- CONTRIBUTING.md | 2 ++ doc/using_uv.md | 67 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 doc/using_uv.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cbd2959e88bd..e17a4a5472ee 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,6 +15,8 @@ We utilize a variety of tools to ensure smooth development, testing, and code qu - Virtualenv: [Virtualenv](https://virtualenv.pypa.io/en/latest/) is leveraged by Tox to create isolated environments for each test suite, ensuring consistent dependencies and reducing conflicts. +- UV: [UV](https://docs.astral.sh/uv/) is a fast package manager that can manage Python versions, run and install Python packages, and be used instead of pip, virtualenv, and more. + - Pytest: [Pytest](https://docs.pytest.org/en/stable/) is the test framework we use for writing and running our unit tests. It supports fixtures, parameterized tests, and other features that make testing more powerful and flexible. - Pylint: [Pylint](https://pylint.readthedocs.io/en/stable/) is used for code linting to enforce coding standards and catch potential issues early. Maintaining a consistent code style is important, and Pylint helps achieve that goal. diff --git a/doc/using_uv.md b/doc/using_uv.md new file mode 100644 index 000000000000..2bd5ff6a8738 --- /dev/null +++ b/doc/using_uv.md @@ -0,0 +1,67 @@ +# hi + +# Pre-requisite + +- You need to have Python installed +- We recommend at least 3.11 to have access to more tool by default (more can be added later) + +- You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically. + +# Initial setup + +- Go to the folder of the package you're working on, for instance `sdk/contoso/azure-contoso` + +## Setup with uv + +`uv venv` + +`uv pip install -r dev_requirements.txt` + +## Setup with pip/python + +`python -m venv .venv` + +`source .venv/bin/activate # for WSL` + +`.venv\Scripts\activate # for Windows` + + +`- pip install -r dev_requirements.txt` + +# Using the CLI + +## With uv + +``` +# uv loads the venv for you when you prefix the command with uvx +uvx azpysdk +``` + +## With venv + +``` +# You can call the command directly if you activated the venv +azpysdk +``` + +# List of available tools + +- mypy +- pylint +- sphinx + +# Advanced scenario + +## Using different Python version + +If you want to locally test different Python versions to check compatibility with all the required Python versions, do that. + +Note that this is optional, and you can rely onn CI to test python versions. + +## With uv + +`uvx --python=3.9 azpysdk mypy` + +## With python + +You need to install pyenv (explain that) \ No newline at end of file From b27ceced067ce89e806ed5e30c982fd48fa21140 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Thu, 4 Sep 2025 14:30:16 -0700 Subject: [PATCH 02/11] revise --- doc/using_uv.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/doc/using_uv.md b/doc/using_uv.md index 2bd5ff6a8738..62fb9f023ec3 100644 --- a/doc/using_uv.md +++ b/doc/using_uv.md @@ -3,8 +3,7 @@ # Pre-requisite - You need to have Python installed -- We recommend at least 3.11 to have access to more tool by default (more can be added later) - +- We recommend at least 3.11 to have access to more tools by default - You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically. # Initial setup @@ -59,9 +58,16 @@ If you want to locally test different Python versions to check compatibility wit Note that this is optional, and you can rely onn CI to test python versions. ## With uv +You can specify a Python version upon venv creation: + +`uvx --python=3.11 azpysdk mypy` -`uvx --python=3.9 azpysdk mypy` +## With Python -## With python +You need to install [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation), which lets you easily switch between multiple versions of Python. -You need to install pyenv (explain that) \ No newline at end of file +To switch Python versions: +``` +pyenv install 3.10 +pyenv global 3.10 +``` \ No newline at end of file From 1e1ff6ffaf5b79c95ac1517d9f3e87585c205b86 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Thu, 4 Sep 2025 14:41:26 -0700 Subject: [PATCH 03/11] minor fix --- doc/using_uv.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/doc/using_uv.md b/doc/using_uv.md index 62fb9f023ec3..b91be74b5142 100644 --- a/doc/using_uv.md +++ b/doc/using_uv.md @@ -3,7 +3,7 @@ # Pre-requisite - You need to have Python installed -- We recommend at least 3.11 to have access to more tools by default +- We recommend at least version 3.11 to have access to more tools by default - You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically. # Initial setup @@ -20,10 +20,13 @@ `python -m venv .venv` -`source .venv/bin/activate # for WSL` - -`.venv\Scripts\activate # for Windows` +``` +# for WSL +source .venv/bin/activate +# for Windows +.venv\Scripts\activate +``` `- pip install -r dev_requirements.txt` From 4b2ebf1674b900fda371a3d5777a763be87fb690 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Thu, 4 Sep 2025 14:50:36 -0700 Subject: [PATCH 04/11] edits --- doc/using_uv.md | 43 ++++++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/doc/using_uv.md b/doc/using_uv.md index b91be74b5142..14bae5b461c9 100644 --- a/doc/using_uv.md +++ b/doc/using_uv.md @@ -1,26 +1,32 @@ -# hi +# Tool Usage Guide -# Pre-requisite +## Available Tools + +- mypy +- pylint +- sphinx + +## Pre-requisite - You need to have Python installed - We recommend at least version 3.11 to have access to more tools by default - You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically. -# Initial setup +## Initial setup - Go to the folder of the package you're working on, for instance `sdk/contoso/azure-contoso` -## Setup with uv +### Setup with uv `uv venv` `uv pip install -r dev_requirements.txt` -## Setup with pip/python +### Setup with pip/python `python -m venv .venv` -``` +```bash # for WSL source .venv/bin/activate @@ -28,44 +34,39 @@ source .venv/bin/activate .venv\Scripts\activate ``` -`- pip install -r dev_requirements.txt` +`pip install -r dev_requirements.txt` -# Using the CLI +## Using the CLI -## With uv +### With uv -``` +```bash # uv loads the venv for you when you prefix the command with uvx uvx azpysdk ``` -## With venv +### With venv ``` # You can call the command directly if you activated the venv azpysdk ``` -# List of available tools +## Advanced scenario -- mypy -- pylint -- sphinx - -# Advanced scenario - -## Using different Python version +### Using different Python version If you want to locally test different Python versions to check compatibility with all the required Python versions, do that. Note that this is optional, and you can rely onn CI to test python versions. -## With uv +#### With uv + You can specify a Python version upon venv creation: `uvx --python=3.11 azpysdk mypy` -## With Python +#### With Python You need to install [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation), which lets you easily switch between multiple versions of Python. From c7eca2662a79fa6b6214976d81c04b5bef93a279 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Thu, 4 Sep 2025 14:53:16 -0700 Subject: [PATCH 05/11] minor edits --- doc/using_uv.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/using_uv.md b/doc/using_uv.md index 14bae5b461c9..c7599043f38a 100644 --- a/doc/using_uv.md +++ b/doc/using_uv.md @@ -47,7 +47,7 @@ uvx azpysdk ### With venv -``` +```bash # You can call the command directly if you activated the venv azpysdk ``` @@ -56,7 +56,7 @@ azpysdk ### Using different Python version -If you want to locally test different Python versions to check compatibility with all the required Python versions, do that. +You can locally test different Python versions to check compatibility with all the required Python versions. Note that this is optional, and you can rely onn CI to test python versions. From ffb60817a0771c37a56587d4f248fa9ac3c38c07 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Fri, 5 Sep 2025 16:04:18 -0700 Subject: [PATCH 06/11] uv venv edit --- doc/{using_uv.md => tool_usage_guide.md} | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) rename doc/{using_uv.md => tool_usage_guide.md} (97%) diff --git a/doc/using_uv.md b/doc/tool_usage_guide.md similarity index 97% rename from doc/using_uv.md rename to doc/tool_usage_guide.md index c7599043f38a..df75c51d246e 100644 --- a/doc/using_uv.md +++ b/doc/tool_usage_guide.md @@ -63,8 +63,7 @@ Note that this is optional, and you can rely onn CI to test python versions. #### With uv You can specify a Python version upon venv creation: - -`uvx --python=3.11 azpysdk mypy` +`uv venv --python 3.11` #### With Python From 22cfbb13db7786aed7209f07a3b2858207eab2dd Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Mon, 8 Sep 2025 15:00:10 -0700 Subject: [PATCH 07/11] revisions from review --- doc/tool_usage_guide.md | 60 ++++++++++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/doc/tool_usage_guide.md b/doc/tool_usage_guide.md index df75c51d246e..d24fdb639268 100644 --- a/doc/tool_usage_guide.md +++ b/doc/tool_usage_guide.md @@ -1,15 +1,52 @@ # Tool Usage Guide +As part of shipping SDKs that can be trusted by customers, the azure-sdk-for-python dev team maintains a suite of `checks` that can be utilized by devs to ensure predictable quality measures that should prevent or allow shipping. These `checks` are implemented in a single entrypoint within local development package `eng/tools/azure-sdk-tools`. This package provides: + + - Templated package generation for mgmt packages + - The test-framework that all packages within this monorepo use (including record/playback integration) + - A ton of CI related tasks and boilerplate + - Static analysis code + +A `tool` in this context is merely a single entrypoint provided by the `azpysdk` entrypoint in the `eng/tools/azure-sdk-tools` package. + ## Available Tools +This repo is currently migrating all checks from a slower `tox`-based framework, to a lightweight implementation that uses `asyncio` to simultaneously run checks. This tools list is the current set that has been migrated from `tox` to the `azpysdk` entrypoint. + - mypy - pylint - sphinx -## Pre-requisite +## Common arguments + +### Globbing +The azpysdk is intended to be used from within the azure-sdk-for-python repository. A user can invoke + +``` +/azure-sdk-for-python/> azpysdk import_all azure-storage* # will run import_all for all packages starting with `azure-storage` + +/azure-sdk-for-python/> azpysdk import_all azure-storage-blob,azure-core # invoke import_all for two packages + +/azure-sdk-for-python/sdk/core/azure-core/> azpysdk import_all . # invoke import_all for the local package only +``` + +### Automatically isolating the environment + +The targeted tool should be able to run in an isolated environment for a couple reasons: +- When attempting to diagnose an issue, sometimes a user is best served by completely wiping their `venv` and starting over from scatch. Issues may stem from having additional deps downloaded that normally wouldn't be installed with the package. +- In `CI`, we _have_ to run in isolated virtual environments for some checks to allow processes like `multiple pytest` runs to invoke without accidentally stomping on each other's progress or hitting file contention errors.. CI passes this flag by default from `dispatch_checks.py`. + +To utilize this feature, add `--isolate` to any `azpysdk` invocation: + +```bash +/> azpysdk import_all azure-storage-blob --isolate +# will install and run within a venv in `sdk/storage/azure-storage-blob/.venv_import_all/ +``` + +## Prerequisite - You need to have Python installed -- We recommend at least version 3.11 to have access to more tools by default +- The monorepo requires a minimum of `python 3.9`, but `>=3.11` is required for the `sphinx` check due to compatibility constraints with external processes. - You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically. ## Initial setup @@ -20,6 +57,14 @@ `uv venv` +```bash +# for WSL +source .venv/bin/activate + +# for Windows +.venv\Scripts\activate +``` + `uv pip install -r dev_requirements.txt` ### Setup with pip/python @@ -38,15 +83,6 @@ source .venv/bin/activate ## Using the CLI -### With uv - -```bash -# uv loads the venv for you when you prefix the command with uvx -uvx azpysdk -``` - -### With venv - ```bash # You can call the command directly if you activated the venv azpysdk @@ -58,7 +94,7 @@ azpysdk You can locally test different Python versions to check compatibility with all the required Python versions. -Note that this is optional, and you can rely onn CI to test python versions. +Note that this is optional, and you can rely on CI to test python versions. #### With uv From f274d4730e2e89af0912b8f19525eb5b5df35bd2 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Mon, 15 Sep 2025 12:07:13 -0700 Subject: [PATCH 08/11] add check table and word to cspell --- .vscode/cspell.json | 1 + doc/tool_usage_guide.md | 10 +++++++--- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.vscode/cspell.json b/.vscode/cspell.json index 4a494888e05a..04ce4630b8a6 100644 --- a/.vscode/cspell.json +++ b/.vscode/cspell.json @@ -197,6 +197,7 @@ "avroencoder", "Avs", "azcmagent", + "azpysdk", "azsdk", "azurecr", "azuremgmtcore", diff --git a/doc/tool_usage_guide.md b/doc/tool_usage_guide.md index d24fdb639268..9ac82d8802f9 100644 --- a/doc/tool_usage_guide.md +++ b/doc/tool_usage_guide.md @@ -13,9 +13,13 @@ A `tool` in this context is merely a single entrypoint provided by the `azpysdk` This repo is currently migrating all checks from a slower `tox`-based framework, to a lightweight implementation that uses `asyncio` to simultaneously run checks. This tools list is the current set that has been migrated from `tox` to the `azpysdk` entrypoint. -- mypy -- pylint -- sphinx +|tool|description|invocation| +|---|---|---| +|`pylint`| Runs `pylint` checks or `pylint-next` checks. (based on presence of `--next` argument) | `azpysdk pylint .` | +|`mypy`| Runs `mypy` checks or `mypy-next` checks. (based on presence of `--next` argument) | `azpysdk mypy --next=True .` | +|`sphinx`| Generates a documentation website for the targeted packages. Runs `sphinx` or `next-sphinx` (based on presence of `--next` argument). | `azpysdk sphinx .` | +|`black`| Runs `black` checks. | `azpysdk black .` | +|`import_all`| Installs the package w/ default dependencies, then attempts to `import *` from the base namespace. Ensures that all imports will resolve after a base install and import. | `azpysdk import_all .` | ## Common arguments From 0cb43fef7ac22a6c3f99bf4b5c973cc91b40ddb3 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Mon, 15 Sep 2025 12:08:42 -0700 Subject: [PATCH 09/11] typo --- doc/tool_usage_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tool_usage_guide.md b/doc/tool_usage_guide.md index 9ac82d8802f9..fd1f22cb122c 100644 --- a/doc/tool_usage_guide.md +++ b/doc/tool_usage_guide.md @@ -15,8 +15,8 @@ This repo is currently migrating all checks from a slower `tox`-based framework, |tool|description|invocation| |---|---|---| -|`pylint`| Runs `pylint` checks or `pylint-next` checks. (based on presence of `--next` argument) | `azpysdk pylint .` | -|`mypy`| Runs `mypy` checks or `mypy-next` checks. (based on presence of `--next` argument) | `azpysdk mypy --next=True .` | +|`pylint`| Runs `pylint` checks or `next-pylint` checks. (based on presence of `--next` argument) | `azpysdk pylint .` | +|`mypy`| Runs `mypy` checks or `next-mypy` checks. (based on presence of `--next` argument) | `azpysdk mypy --next=True .` | |`sphinx`| Generates a documentation website for the targeted packages. Runs `sphinx` or `next-sphinx` (based on presence of `--next` argument). | `azpysdk sphinx .` | |`black`| Runs `black` checks. | `azpysdk black .` | |`import_all`| Installs the package w/ default dependencies, then attempts to `import *` from the base namespace. Ensures that all imports will resolve after a base install and import. | `azpysdk import_all .` | From eaf40f7fbbdac97a438c2921e84923b726a7a4b8 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Mon, 15 Sep 2025 12:16:57 -0700 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- doc/tool_usage_guide.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/tool_usage_guide.md b/doc/tool_usage_guide.md index fd1f22cb122c..5ac7ad8a05bb 100644 --- a/doc/tool_usage_guide.md +++ b/doc/tool_usage_guide.md @@ -1,6 +1,6 @@ # Tool Usage Guide -As part of shipping SDKs that can be trusted by customers, the azure-sdk-for-python dev team maintains a suite of `checks` that can be utilized by devs to ensure predictable quality measures that should prevent or allow shipping. These `checks` are implemented in a single entrypoint within local development package `eng/tools/azure-sdk-tools`. This package provides: +As part of shipping SDKs that can be trusted by customers, the azure-sdk-for-python dev team maintains a suite of `checks` that can be utilized by developers to ensure predictable quality measures. These checks help determine whether a package meets the required quality standards for release or needs further work before it can be shipped. These `checks` are implemented in a single entrypoint within the local development package `eng/tools/azure-sdk-tools`. This package provides: - Templated package generation for mgmt packages - The test-framework that all packages within this monorepo use (including record/playback integration) @@ -37,8 +37,8 @@ The azpysdk is intended to be used from within the azure-sdk-for-python reposito ### Automatically isolating the environment The targeted tool should be able to run in an isolated environment for a couple reasons: -- When attempting to diagnose an issue, sometimes a user is best served by completely wiping their `venv` and starting over from scatch. Issues may stem from having additional deps downloaded that normally wouldn't be installed with the package. -- In `CI`, we _have_ to run in isolated virtual environments for some checks to allow processes like `multiple pytest` runs to invoke without accidentally stomping on each other's progress or hitting file contention errors.. CI passes this flag by default from `dispatch_checks.py`. +- When attempting to diagnose an issue, sometimes a user is best served by completely wiping their `venv` and starting over from scratch. Issues may stem from having additional deps downloaded that normally wouldn't be installed with the package. +- In `CI`, we _have_ to run in isolated virtual environments for some checks to allow processes like `multiple pytest` runs to invoke without accidentally stomping on each other's progress or hitting file contention errors. CI passes this flag by default from `dispatch_checks.py`. To utilize this feature, add `--isolate` to any `azpysdk` invocation: From dd87ca676e6091330b2570697605f54c16d53899 Mon Sep 17 00:00:00 2001 From: jenny <63012604+JennyPng@users.noreply.github.com> Date: Tue, 16 Sep 2025 09:47:46 -0700 Subject: [PATCH 11/11] Update .vscode/cspell.json Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com> --- .vscode/cspell.json | 1 + 1 file changed, 1 insertion(+) diff --git a/.vscode/cspell.json b/.vscode/cspell.json index 04ce4630b8a6..4abb2c97d208 100644 --- a/.vscode/cspell.json +++ b/.vscode/cspell.json @@ -198,6 +198,7 @@ "Avs", "azcmagent", "azpysdk", + "pyenv", "azsdk", "azurecr", "azuremgmtcore",