-
-
Notifications
You must be signed in to change notification settings - Fork 34.1k
gh-123152: Add a Concurrency Howto Page #123163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
ericsnowcurrently
wants to merge
81
commits into
python:main
Choose a base branch
from
ericsnowcurrently:concurrency-howto
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit
Hold shift + click to select a range
4187872
Let "python3" be set at the commandline.
ericsnowcurrently 424dc37
Add a NEWS entry.
ericsnowcurrently 9ab68fa
Add the concurrency howto doc.
ericsnowcurrently fce455e
Add more explanation.
ericsnowcurrently 28d8ac0
More explanation.
ericsnowcurrently 503d829
Re-structure the comparison table.
ericsnowcurrently e5d904f
Add basic examples for each concurrency model.
ericsnowcurrently b7bb9a5
Clarify about "spook action at a distance".
ericsnowcurrently 7efb9b4
Clarify about the "pain" threads can cause.
ericsnowcurrently daaa02b
Add references to the howto in the library docs.
ericsnowcurrently ad50bdd
Clean up the Python concurrency models section.
ericsnowcurrently 3a82978
Add a section about concurrent.futures.
ericsnowcurrently 560f26e
Make the examples runnable.
ericsnowcurrently 4f49ed7
Revert changes to Makefile.
ericsnowcurrently 5151e3b
Fix a typo.
ericsnowcurrently 14e59f8
Fix sphinx warnings.
ericsnowcurrently 0b27fd9
Adjust the references.
ericsnowcurrently 0586c72
Fill out and restructure the comparisons.
ericsnowcurrently 8df61ad
Fill out the section about Python theads.
ericsnowcurrently 269eaac
Fill out the section about multiple interpreters.
ericsnowcurrently 7754912
Fill out the section about multprocessing.
ericsnowcurrently 2c9a793
Fill out the section about async/await.
ericsnowcurrently 2975bbb
Shuffle the concurrency model order.
ericsnowcurrently b82a18b
Fill out the section about distributed concurrency.
ericsnowcurrently 5e7f7c3
Drop an incomplete note.
ericsnowcurrently c06b843
Fix typos.
ericsnowcurrently e05e76e
Add a missing divider.
ericsnowcurrently e7144ec
Add a link to the "overhead" table.
ericsnowcurrently e131b83
Expand the general explanation of workloads.
ericsnowcurrently c601e27
Updates for the first two workload examples.
ericsnowcurrently 5c05581
Do not worry about demonstrating a web service.
ericsnowcurrently a0a8ebb
Move the grep example up.
ericsnowcurrently c3d1688
Implement threads for the grep example.
ericsnowcurrently 48ac79d
Implement grep using concurrent.futures.
ericsnowcurrently 9a1a81c
Fix nested lists and add quadrants.
ericsnowcurrently 03156b5
Implement grep using multiprocessing.
ericsnowcurrently e0d833e
Start reorganizing, and dump a bunch of explanation.
ericsnowcurrently 6dd91e3
Clarify about Python-supported concurrency models.
ericsnowcurrently 1e935df
Small clarifications and cleanup.
ericsnowcurrently 4fdf9f5
Move the high-level APIs section down.
ericsnowcurrently cb843f6
Clarify about the comparisons.
ericsnowcurrently 8a87617
Updates about free-threading caveats.
ericsnowcurrently 288c4f4
Formatting and links for intro section.
ericsnowcurrently 0b768ae
wording tweaks
ericsnowcurrently ab24e83
Update the section about multiple interpreters.
ericsnowcurrently f7f1769
Update the section about coroutines.
ericsnowcurrently fdf46cb
Update the sections about multi-processing.
ericsnowcurrently 4cc62a1
Clear out the section on downsides.
ericsnowcurrently 54bacb6
Update the section about shared resources.
ericsnowcurrently 49ec62f
Normalize TODO comments.
ericsnowcurrently 8fd8406
Move the explanations to the right places.
ericsnowcurrently fdcc128
Add sub-sections.
ericsnowcurrently 3033875
Move the caveats subsections around.
ericsnowcurrently 3e4ab6a
Fix formatting.
ericsnowcurrently f07c696
Drop a TODO.
ericsnowcurrently 7a04361
Hide the incomplete portions of text.
ericsnowcurrently f9815bb
Tweak a table header.
ericsnowcurrently 48551a7
Fix a table header.
ericsnowcurrently eac23f9
Fix the sidebar table of contents.
ericsnowcurrently 3e1c53e
Expand the grep section and code.
ericsnowcurrently 9f6ea1f
Make the examples more uniform.
ericsnowcurrently 3aaa4d1
Fix some wording.
ericsnowcurrently 3a7b0be
Reorganize the example code a little.
ericsnowcurrently 36edcce
Split up the examples file.
ericsnowcurrently 7b4af12
Tweak CLI.
ericsnowcurrently bc6e725
Simplify the examples.
ericsnowcurrently f4749f6
Drop all the context manager stuff.
ericsnowcurrently eaaecca
Implement the async example.
ericsnowcurrently ebf3130
Drop the text about do_search().
ericsnowcurrently 21ff9e1
Fix a typo.
ericsnowcurrently 3682348
Move the grep examples to individual files.
ericsnowcurrently 10ee369
Tweak run-examples.py to run the newer example code.
ericsnowcurrently e6f0088
Drop the combined grep package.
ericsnowcurrently 3f2637b
Fix the highlighting.
ericsnowcurrently 9ce6ccf
cleanups
ericsnowcurrently 10a6571
Fix typos and wording.
ericsnowcurrently 793bfa0
Limit side-by-side example lines to 60 characters.
ericsnowcurrently 3e19d7e
Fix the examples.
ericsnowcurrently f826c9c
Fix a ref.
ericsnowcurrently 049f1de
lint
ericsnowcurrently 7686e10
Merge branch 'main' into concurrency-howto
AA-Turner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Move the high-level APIs section down.
- Loading branch information
commit 4fdf9f511fc1f6c33214f45b12dc06191731938b
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -388,43 +388,6 @@ exposure | |
| - medium? | ||
| - n/a | ||
|
|
||
| high-level APIs | ||
| --------------- | ||
|
|
||
| Also note that Python's stdlib provides various higher-level APIs | ||
| that support these concurrency models in various contexts: | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
| :class: borderless vert-aligned | ||
| :align: left | ||
|
|
||
| * - | ||
| - :mod:`concurrent.futures` | ||
| - :mod:`socketserver` | ||
| - :mod:`http.server` | ||
| * - free-threading | ||
| - :class:`yes <concurrent.futures.ThreadPoolExecutor>` | ||
| - :class:`yes <socketserver.ThreadingMixIn>` | ||
| - :class:`yes <http.server.ThreadingHTTPServer>` | ||
| * - multiple interpreters | ||
| - (`pending <python-stdlib-interpreters_>`_) | ||
| - | ||
| - | ||
| * - coroutines | ||
| - ??? | ||
| - | ||
| - | ||
| * - multi-processing | ||
| - | :class:`yes <concurrent.futures.ProcessPoolExecutor>` | ||
| | (:class:`similar <multiprocessing.pool.Pool>`) | ||
| - :class:`yes <socketserver.ForkingMixIn>` | ||
| - | ||
| * - distributed | ||
| - ??? | ||
| - | ||
| - | ||
|
|
||
| Critical caveats | ||
| ---------------- | ||
|
|
||
|
|
@@ -746,6 +709,46 @@ was started is valuable when debugging, as is knowing where a callback | |
| was registered. | ||
|
|
||
|
|
||
|
|
||
|
|
||
| high-level APIs | ||
| --------------- | ||
|
|
||
| Also note that Python's stdlib provides various higher-level APIs | ||
| that support these concurrency models in various contexts: | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
| :class: borderless vert-aligned | ||
| :align: left | ||
|
|
||
| * - | ||
| - :mod:`concurrent.futures` | ||
| - :mod:`socketserver` | ||
| - :mod:`http.server` | ||
| * - free-threading | ||
| - :class:`yes <concurrent.futures.ThreadPoolExecutor>` | ||
| - :class:`yes <socketserver.ThreadingMixIn>` | ||
| - :class:`yes <http.server.ThreadingHTTPServer>` | ||
| * - multiple interpreters | ||
| - (`pending <python-stdlib-interpreters_>`_) | ||
| - | ||
| - | ||
| * - coroutines | ||
| - ??? | ||
| - | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| - | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| * - multi-processing | ||
| - | :class:`yes <concurrent.futures.ProcessPoolExecutor>` | ||
| | (:class:`similar <multiprocessing.pool.Pool>`) | ||
| - :class:`yes <socketserver.ForkingMixIn>` | ||
| - | ||
| * - distributed | ||
| - ??? | ||
| - | ||
| - | ||
|
|
||
|
|
||
| Designing A Program For Concurrency | ||
| =================================== | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe
gather? https://docs.python.org/dev/library/asyncio-task.html#asyncio.gather