Micro-optimization by asvetlov · Pull Request #3095 · aio-libs/aiohttp

asvetlov · 2018-06-19T12:35:39Z

No description provided.

kxepal · 2018-06-19T12:39:31Z

aiohttp/web_protocol.py

+                 '_waiter', '_error_handler', '_task_handler',
+                 '_upgrade', '_payload_parser', '_request_parser',
+                 '_reading_paused', 'logger', 'debug', 'access_log',
+                 'access_logger', '_close', '_force_close')


May be have it one-item-per-line and sorted? Would simplify management and new items addition if that happens.

__slots__ = ( '_close', '_error_handler', '_force_close', '_keep_alive', '_keepalive_handle', '_keepalive_time', '_keepalive_timeout', '_lingering_time', '_manager', '_message_tail', '_messages', '_payload_parser', '_reading_paused', '_request_count', '_request_factory', '_request_handler', '_request_parser', '_task_handler', '_tcp_keepalive', '_upgrade', '_waiter', 'access_log', 'access_logger', 'debug', 'logger', )

Sorting is boring and weak: very easy to make a mistake in sort order.

I'm thinking about ultimately applying black but not ready for it yet.

black is good, but you wouldn't like what it will do with string quotes - it prefers double ultimately.

Maybe we can live with it.

What is black?

https://github.com/ambv/black

@kxepal there's a setting to disable quotes normalization. But there's other things I dislike about the style of blackened code.

Yeah, that's a new feature. I agree that black isn't perfect, but eventually, there are no much good alternatives. Good news: it's quite easy to fork it and fix stuff to make you like it. In the end, we could have some aio-libs style guide which is backed by ours formatter. Still that's better than nothing.

codecov-io · 2018-06-19T15:33:15Z

Codecov Report

Merging #3095 into master will increase coverage by 0.24%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3095      +/-   ##
==========================================
+ Coverage   98.01%   98.25%   +0.24%     
==========================================
  Files          43       43              
  Lines        7823     8275     +452     
  Branches     1352     1525     +173     
==========================================
+ Hits         7668     8131     +463     
+ Misses         56       53       -3     
+ Partials       99       91       -8

Impacted Files	Coverage Δ
aiohttp/hdrs.py	`100% <100%> (ø)`	⬆️
aiohttp/web_exceptions.py	`100% <100%> (ø)`	⬆️
aiohttp/web_protocol.py	`91.75% <100%> (+0.02%)`	⬆️
aiohttp/web_request.py	`99.24% <100%> (+0.57%)`	⬆️
aiohttp/http_parser.py	`98.06% <100%> (ø)`	⬆️
aiohttp/web_urldispatcher.py	`99.11% <100%> (ø)`	⬆️
aiohttp/base_protocol.py	`100% <100%> (ø)`	⬆️
aiohttp/typedefs.py	`100% <0%> (ø)`	⬆️
aiohttp/multipart.py	`96.01% <0%> (+0.54%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d5bb46...552c0b3. Read the comment docs.

kxepal · 2018-06-21T14:25:28Z

@asvetlov How much boost these optimizations gives?

asvetlov · 2018-06-21T15:09:16Z

Not so much on macro benchmarks (4-5%), but HTTP parser is 2 times faster now.

kxepal · 2018-06-21T15:36:44Z

Cool!

asvetlov · 2018-06-21T17:48:56Z

You know, boosting existing pre-optimized code is a hard job.
I see several points for speed up. No one is fascinating but the cummulative effect could be very visible.

================== Features -------- - Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_) - Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_) - Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_) - Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_) - Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_) - Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_) - Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_) - The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_) Bugfixes -------- - Prevent double closing when client connection is released before the last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_) - Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_) - Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_) - Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_) - When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_) - fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_) - correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_) - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - raise 413 "Payload Too Large" rather than raising ValueError in request.post() Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_) - Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_) - Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_) - Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_) - Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_) Improved Documentation ---------------------- - Add a new FAQ entry that clarifies that you should not reuse response objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_) - Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_) - Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_) - Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_) - Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_) - Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_) - Mention that ``app.make_handler`` is deprecated, recommend to use runners API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_) Deprecations and Removals ------------------------- - Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_) - Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_)

This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.2** to **3.4.4**. <details> <summary>Changelog</summary> ### 3.4.4 ``` ================== - Fix installation from sources when compiling toolkit is not available (`3241 <https://github.com/aio-libs/aiohttp/pull/3241>`_) ``` ### 3.4.3 ``` ================== - Add ``app.pre_frozen`` state to properly handle startup signals in sub-applications. (`3237 <https://github.com/aio-libs/aiohttp/pull/3237>`_) ``` ### 3.4.2 ``` ================== - Fix ``iter_chunks`` type annotation (`3230 <https://github.com/aio-libs/aiohttp/pull/3230>`_) ``` ### 3.4.1 ``` ================== - Fix empty header parsing regression. (`3218 <https://github.com/aio-libs/aiohttp/pull/3218>`_) - Fix BaseRequest.raw_headers doc. (`3215 <https://github.com/aio-libs/aiohttp/pull/3215>`_) - Fix documentation building on ReadTheDocs (`3221 <https://github.com/aio-libs/aiohttp/pull/3221>`_) ``` ### 3.4.0 ``` ================== Features -------- - Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_) - Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_) - Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_) - Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_) - Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_) - Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_) - Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_) - The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_) Bugfixes -------- - Prevent double closing when client connection is released before the last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_) - Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_) - Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_) - Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_) - When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_) - fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_) - correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_) - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - raise 413 "Payload Too Large" rather than raising ValueError in request.post() Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_) - Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_) - Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_) - Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_) - Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_) Improved Documentation ---------------------- - Add a new FAQ entry that clarifies that you should not reuse response objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_) - Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_) - Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_) - Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_) - Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_) - Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_) - Mention that ``app.make_handler`` is deprecated, recommend to use runners API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_) Deprecations and Removals ------------------------- - Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_) - Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/aiohttp - Changelog: https://pyup.io/changelogs/aiohttp/ - Repo: https://github.com/aio-libs/aiohttp </details>

lock · 2019-10-28T03:04:01Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new

Micro-optimization

8278dec

kxepal reviewed Jun 19, 2018

View reviewed changes

asvetlov added 2 commits June 19, 2018 16:55

Use faster attr check instead of isinstance

517ffe1

Add changelog

3d72768

asvetlov added 14 commits June 19, 2018 21:52

Optimize Cython parser

e2e2fd2

Improve cythonized parser

1927fd6

Resize bytearray fast

530062a

Fix tests

8c39222

Implement fast bytearray extending

a5d5ec4

Move forward

8a28113

Drop redundant method

b8325eb

Precalculate content-encoding

356b335

Speedup HTTP method lookup

c26d616

Reraise compiler error

e5cebbd

Use alternative strncasecmp definition

8fb1189

Optimize more

904a5f0

Move on

eb26e66

Process header case insensitively

87f1dcb

Remove redundant type check

f8fa6a6

asvetlov added 4 commits June 21, 2018 20:51

Improve generated code formatting

bf2c422

Improve code generator

eba32bb

Improve formatting

781e8d4

Title-case headers

552c0b3

asvetlov merged commit 28f1519 into master Jun 22, 2018

asvetlov deleted the optimize branch June 22, 2018 12:21

germanjoey mentioned this pull request Jul 16, 2018

aiohttp-master failing to build on macOS High Sierra 10.13.5. #3138

Closed

ghost mentioned this pull request Aug 26, 2018

aiohttp versions available: 3.4.0 Harmon758/Harmonbot#301

Closed

das7pad mentioned this pull request Nov 22, 2018

Header cleanup #3404

Merged

5 tasks

cjrh mentioned this pull request Mar 10, 2019

BUG, EASY, PY38: web_protocol.RequestHandler mismatch _keepalive field with __slots__ #3644

Closed

atemate mentioned this pull request May 7, 2019

Fix tests that don't throw DeprecationWarning #3738

Merged

5 tasks

lock bot added the outdated label Oct 28, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 28, 2019

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Oct 28, 2019

Uh oh!

Conversation

asvetlov commented Jun 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Jun 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kxepal commented Jun 21, 2018

Uh oh!

asvetlov commented Jun 21, 2018

Uh oh!

kxepal commented Jun 21, 2018

Uh oh!

asvetlov commented Jun 21, 2018

Uh oh!

lock bot commented Oct 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-io commented Jun 19, 2018 •

edited

Loading