MDEV-29445: Reimplement SET GLOBAL innodb_buffer_pool_size - #3819
Closed
dr-m wants to merge 1 commit into
Closed
Conversation
We deprecate and ignore the parameter innodb_buffer_pool_chunk_size and let the buffer pool size to be changed in arbitrary 1-megabyte increments, all the way up to innodb_buffer_pool_size_max, which must be specified at startup. If innodb_buffer_pool_size_max is not specified, it will default to twice the specified innodb_buffer_pool_size. The buffer pool will be mapped in a contiguous memory area that will be aligned and partitioned into extents of 8 MiB on 64-bit systems and 2 MiB on 32-bit systems. Within an extent, the first few innodb_page_size blocks contain buf_block_t objects that will cover the page frames in the rest of the extent. In this way, there is a trivial mapping between page frames and block descriptors and we do not need any lookup tables like buf_pool.zip_hash or buf_pool_t::chunk_t::map. We will always allocate the same number of block descriptors for an extent, even if we do not need all the buf_block_t in the last extent in case the innodb_buffer_pool_size is not an integer multiple of the of extents size. The minimum innodb_buffer_pool_size is 256*5/4 pages. At the default innodb_page_size=16k this corresponds to 5 MiB. However, now that the innodb_buffer_pool_size includes the memory allocated for the block descriptors, the minimum would be innodb_buffer_pool_size=6m. Innodb_buffer_pool_resize_status: Remove. We will execute buf_pool_t::resize() synchronously in the thread that is executing SET GLOBAL innodb_buffer_pool_size. That operation will run until it completes, or until a KILL statement is executed, the client is disconnected, the buf_flush_page_cleaner() thread notices that we are running out of memory, or the server is shut down. my_large_virtual_alloc(): A new function, similar to my_large_malloc(). On Microsoft Windows, buffer pool resizing is disabled if large pages are being used. buf_pool_t::create(), buf_pool_t::chunk_t::create(): Only initialize the first page descriptor of each chunk. buf_pool_t::lazy_allocate(): Lazily initialize a previously allocated page descriptor and increase buf_pool.n_blocks, which must be below buf_pool.n_blocks_alloc. buf_pool_t::allocate(): Renamed from buf_LRU_get_free_only(). buf_pool_t::LRU_warned: Changed to Atomic_relaxed<bool>, only to be modified by the buf_flush_page_cleaner() thread. buf_pool_t::LRU_shrink(): Check if buffer pool shrinking needs to process a buffer page. buf_pool_t::resize(): Always zero out b->page.zip.data. Failure to do so would cause crashes or corruption in the test innodb.innodb_buffer_pool_resize due to duplicated allocation in the buddy system. Before tarting to shrink the buffer pool, run one batch of buf_flush_page_cleaner() in order to prevent LRU_warn(). Abort shrinking if the buf_flush_page_cleaner() has LRU_warned. buf_pool_t::first_to_withdraw: The first block descriptor that is out of the bounds of the shrunk buffer pool. buf_pool_t::withdrawn: The list of withdrawn blocks. If buf_pool_t::resize() is aborted, we must be able to resurrect the withdrawn blocks in the free list. buf_pool_t::contains_zip(): Added a parameter for the number of least significant pointer bits to disregard, so that we can find any pointers to within a block that is supposed to be free. buf_pool_t::get_info(): Replaces buf_stats_get_pool_info(). innodb_init_param(): Refactored. We must first compute srv_page_size_shift and then determine the valid bounds of innodb_buffer_pool_size. buf_buddy_shrink(): Replaces buf_buddy_realloc(). Part of the work is deferred to buf_buddy_condense_free(), which is being executed when we are not holding any buf_pool.page_hash latch. buf_buddy_condense_free(): Do not relocate blocks. buf_buddy_free_low(): Do not care about buffer pool shrinking. This will be handled by buf_buddy_shrink() and buf_buddy_condense_free(). buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip() when we are allocating from the binary buddy system. Previously we were asserting this on multiple recursion levels. buf_buddy_block_free(), buf_buddy_free_low(): Assert !buf_pool.contains_zip(). buf_buddy_alloc_from(): Remove the redundant parameter j. buf_flush_LRU_list_batch(): Add the parameter shrinking. If we are shrinking, invoke buf_pool_t::LRU_shrink() to see if we must keep going. buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch() if we are shrinking the buffer pool. In that case, we want to minimize the page relocations and just finish as quickly as possible. trx_purge_attach_undo_recs(): Limit purge_sys.n_pages_handled() in every iteration, in case the buffer pool is being shrunk in the middle of a purge batch.
|
|
Contributor
Author
|
This was closed in #3826. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is a rebase of #3107 to the
mainbranch, resolving conflicts related to the removal of the change buffer and the refactoring of the adaptive hash index.The buffer pool will be mapped in a contiguous memory area that will be aligned and divided to extents of 8 MiB on 64-bit systems and 2 MiB on 32-bit systems.
Within an extent, the first few
innodb_page_size pagescontainbuf_block_tobjects that will cover the page frames in the restof the extent. In this way, there is a trivial mapping between page frames and block descriptors and we do not need any lookup tables like
buf_pool.zip_hashorbuf_pool_t::chunk_t::map.We will always allocate the same number of page frames for an extent, even if we do not need all the
buf_block_tin the last extent in case theinnodb_buffer_pool_sizeis not an integer multiple of the of extent size.my_large_virtual_alloc(): A new function, similar tomy_large_malloc().On Microsoft Windows, buffer pool resizing is disabled if large pages are being used.
buf_pool_t::create(): Only initialize the first page descriptor of each chunk.buf_pool_t::lazy_allocate(): Lazily initialize a previously allocated page descriptor and increasebuf_pool.n_blocks, which must be belowbuf_pool.n_blocks_alloc.innodb_init_param(): Refactored. We must first validateinnodb_page_sizeand then determine the valid bounds ofinnodb_buffer_pool_size.Release Notes
We deprecate and ignore the parameter
innodb_buffer_pool_chunk_sizeand let the buffer pool size to be changed in arbitrary 1-megabyte increments, all the way up toinnodb_buffer_pool_size_max, which must be specified at startup.If
innodb_buffer_pool_size_maxis not specified, it will default to twice the specifiedinnodb_buffer_pool_size.The minimum
innodb_buffer_pool_sizeis 320 pages. At the defaultinnodb_page_size=16kthis corresponds to 5 MiB. However, now that theinnodb_buffer_pool_sizeincludes the memory allocated for the block descriptors, the minimum would now beinnodb_buffer_pool_size=6m.Innodb_buffer_pool_resize_statuswill be removed. TheSET GLOBAL innodb_buffer_pool_sizeoperation will block until the buffer pool has been resized or the operation aborted byKILL,SHUTDOWNor disconnect.How can this PR be tested?
This is mostly covered by the regression test suite.
Large-page allocation needs to be tested on Linux and Microsoft Windows. Stress testing with lots of buffer pool resizing during the workload would be helpful.
Basing the PR against the correct MariaDB version
PR quality check