MDEV-35049: Improve adaptive hash index scalability - #3562
Conversation
|
|
|
Sorry, this needs some more work: However, this should not prevent any performance testing. |
|
Thanks to @montywi for pointing out my mistake: all updates of ./mtr --mem --parallel=5 --suite=innodb --mysqld=--loose-innodb-adaptive-hash-index=on |
This turned out to be an independent bug, which I fixed in cc70ca7. |
|
994a740 reverts another attempt to use lock upgrade again. It would lead to occasional deadlocks when running |
377ab11 to
a793977
Compare
13d590e to
d4b37ae
Compare
f3b7361 to
6f63f11
Compare
3fde097 to
2e3fe2f
Compare
|
The revert 36a8b44 was an attempt to see if it would fix a race condition between buffer pool resizing and the adaptive hash index. Unfortunately, while 2 runs of the mandatory amd64-debian-11-msan-clang-16 builder were successful, the 3rd attempt reproduced the assertion failures in |
dr-m
left a comment
There was a problem hiding this comment.
It looks like I figured out the cause of some trouble. Hopefully addressing these will allow some stress tests to pass.
| /* Now that we are holding block->page.lock, the page cannot be | ||
| modified or evicted (buf_page_t::can_relocate() will not hold) while | ||
| we validate the guessed rec. Thus, releasing part.latch is safe. */ | ||
| part.latch.rd_unlock(); | ||
|
|
||
| while (const rec_t* next_rec = page_rec_get_next_const(rec)) { | ||
| if (page_rec_is_supremum(next_rec)) { | ||
| const auto state= block->page.state(); | ||
|
|
||
| if (!left_side) { | ||
| if (UNIV_UNLIKELY(state < buf_page_t::UNFIXED)) |
There was a problem hiding this comment.
It is actually not safe to invoke part.latch.rd_unlock() before validating also state and block->index. Immediately after reading those, it is safe. At this point of time, we do not even know if the AHI entry is for our index.
There was a problem hiding this comment.
I got some further thoughts regarding this. What if instead of just reading block->page.state() we invoked block->page.fix() already here? We will be doing that a little later anyway, unless the state was buf_page_t::REMOVE_HASH. The only question is how to make sure that a possible concurrent execution of buf_page_t::REMOVE_HASH will not run into any trouble. A possible solution could be to protect also this increment under the buf_pool.page_hash latch, so that we can back off without damage if we had incremented buf_page_t::REMOVE_HASH to buf_page_t::FREED. If the buf_page_t::lock and the buf_page_t::state() field are located in different cache line, this would seem to be a bad idea: adding 1 more cache line access (a read-modify-write access at that) and conditional branch inside the critical section of a buf_pool.page_hash latch.
| const rec_t *rec= node->rec; | ||
| buf_block_t *block= buf_pool.block_from_ahi(rec); | ||
| #if defined UNIV_AHI_DEBUG || defined UNIV_DEBUG | ||
| ut_a(block == node->block); | ||
| #endif |
There was a problem hiding this comment.
This logic here is unchanged; the code was merely reformatted.
There is a yet undiagnosed race condition with buffer pool resizing, which I filed as MDEV-35485. I will implement a temporary work-around (replace the buf_pool.block_from_ahi(rec) with node->block) in the debug instrumented builds, so that we can test this a bit more with buffer pool resizing. With that work-around and the deferred part.latch.rd_unlock(), the test innodb.innodb_buffer_pool_resize passed 10×140 rounds in a cmake -DWITH_MSAN=ON build. It would typically crash on the 1st or 2nd test round, either due to ut_a(offs < chunk->size) or some assertion related to block->page.state or block->page.lock.
36a8b44 to
b55973e
Compare
There was a problem hiding this comment.
I did not find obvious errors during code review. Fixed-size allocator, crc32-c as hash function for hash index, hashing contiguous array of bytes instead of separate fields, part.blocks_mutex along with part.spare and other changes should improve hash index performance.
877d75a to
0e1ff55
Compare
2064cc8 to
500a79f
Compare
Let us use implement a simple fixed-size allocator for the adaptive hash index, insted of complicating mem_heap_t or mem_block_info_t. MEM_HEAP_BTR_SEARCH: Remove. mem_block_info_t::free_block(), mem_heap_free_block_free(): Remove. mem_heap_free_top(), mem_heap_get_top(): Remove. btr_sea::partition::spare: Replaces mem_block_info_t::free_block. This keeps one spare block per adaptive hash index partition, to process an insert. We must not wait for buf_pool.mutex while holding any btr_sea::partition::latch. That is why we cache one block for future allocations. This is protected by a new btr_sea::partition::blocks_mutex in order to relieve pressure on btr_sea::partition::latch. btr_sea::partition::prepare_insert(): Replaces btr_search_check_free_space_in_heap(). btr_sea::partition::erase(): Replaces ha_search_and_delete_if_found(). btr_sea::partition::cleanup_after_erase(): Replaces the most part of ha_delete_hash_node(). Unlike the previous implementation, we will retain a spare block for prepare_insert(). This should reduce some contention on buf_pool.mutex. btr_search.n_parts: Replaces btr_ahi_parts. btr_search.enabled: Replaces btr_search_enabled. This must hold whenever buf_block_t::index is set while a thread is holding a btr_sea::partition::latch. dict_index_t::search_info: Remove pointer indirection, and use Atomic_relaxed or Atomic_counter for most fields. btr_search_guess_on_hash(): Let the caller ensure that latch_mode is BTR_MODIFY_LEAF or BTR_SEARCH_LEAF. Release btr_sea::partition::latch before buffer-fixing the block. The page latch that we already acquired is preventing buffer pool eviction. We must validate both block->index and block->page.state while holding part.latch in order to avoid race conditions with buffer page relocation or buf_pool_t::resize(). btr_search_check_guess(): Remove the constant parameter can_only_compare_to_cursor_rec=false. ahi_node: Replaces ha_node_t. This has been tested by running the regression test suite with the adaptive hash index enabled: ./mtr --mysqld=--loose-innodb-adaptive-hash-index=ON Reviewed by: Vladislav Lesin
For the adaptive hash index, dtuple_fold() and rec_fold() were employing
a slow rolling hash algorithm, computing hash values ("fold") for one
field and one byte at a time, while depending on calls to
rec_get_offsets().
We already have optimized implementations of CRC-32C and have been
successfully using that function in some other InnoDB tables, but not
yet in the adaptive hash index.
Any linear function such as any CRC will fail the avalanche test that
any cryptographically secure hash function is expected to pass:
any single-bit change in the input key should affect on average half
the bits in the output.
But we always were happy with less than cryptographically secure:
in fact, ut_fold_ulint_pair() or ut_fold_binary() are just about as
linear as any CRC, using a combination of multiplication and addition,
partly carry-less. It is worth noting that exclusive-or corresponds to
carry-less subtraction or addition in a binary Galois field, or GF(2).
We only need some way of reducing key prefixes into hash values.
The CRC-32C should be better than a Rabin–Karp rolling hash algorithm.
Compared to the old hash algorithm, it has the drawback that there will
be only 32 bits of entropy before we choose the hash table cell by a
modulus operation. The size of each adaptive hash index array is
(innodb_buffer_pool_size / 512) / innodb_adaptive_hash_index_parts.
With the maximum number of partitions (512), we would not exceed 1<<32
elements per array until the buffer pool size exceeds 1<<50 bytes (1 PiB).
We would hit other limits before that: the virtual address space on many
contemporary 64-bit processor implementations is only 48 bits (256 TiB).
So, we can simply go for the SIMD accelerated CRC-32C.
rec_fold(): Take a combined parameter n_bytes_fields. Determine the
length of each field on the fly, and compute CRC-32C over a single
contiguous range of bytes, from the start of the record payload area
to the end of the last full or partial field. For secondary index records
in ROW_FORMAT=REDUNDANT, also the data area that is reserved for NULL
values (to facilitate in-place updates between NULL and NOT NULL values)
will be included in the count. Luckily, InnoDB always zero-initialized
such unused area; refer to data_write_sql_null() in
rec_convert_dtuple_to_rec_old(). For other than ROW_FORMAT=REDUNDANT,
no space is allocated for NULL values, and therefore the CRC-32C will
only cover the actual payload of the key prefix.
dtuple_fold(): For ROW_FORMAT=REDUNDANT, include the dummy NULL values
in the CRC-32C, so that the values will be comparable with rec_fold().
innodb_ahi-t: A unit test for rec_fold() and dtuple_fold().
btr_search_build_page_hash_index(), btr_search_drop_page_hash_index():
Use a fixed-size stack buffer for computing the fold values, to avoid
dynamic memory allocation.
btr_search_drop_page_hash_index(): Do not release part.latch if we
need to invoke multiple batches of rec_fold().
dtuple_t: Allocate fewer bits for the fields. The maximum number of
data fields is about 1023, so uint16_t will be fine for them. The
info_bits is stored in less than 1 byte.
ut_pair_min(), ut_pair_cmp(): Remove. We can actually combine and compare
int(n_fields << 16 | n_bytes).
PAGE_CUR_LE_OR_EXTENDS, PAGE_CUR_DBG: Remove. These were never defined,
because they would only work with latin1_swedish_ci if at all.
btr_cur_t::check_mismatch(): Replaces !btr_search_check_guess().
cmp_dtuple_rec_bytes(): Replaces cmp_dtuple_rec_with_match_bytes().
Determine the offsets of fields on the fly.
page_cur_try_search_shortcut_bytes(): This caller of
cmp_dtuple_rec_bytes() will not be invoked on the change buffer tree.
cmp_dtuple_rec_leaf(): Replaces cmp_dtuple_rec_with_match()
for comparing leaf-page records.
buf_block_t::ahi_left_bytes_fields: Consolidated Atomic_relaxed<uint32_t>
of curr_left_side << 31 | curr_n_bytes << 16 | curr_n_fields.
The other set of parameters (n_fields, n_bytes, left_side) was removed
as redundant.
btr_search_update_hash_node_on_insert(): Merged to
btr_search_update_hash_on_insert().
btr_search_build_page_hash_index(): Take combined left_bytes_fields
instead of n_fields, n_bytes, left_side.
btr_search_update_block_hash_info(), btr_search_update_hash_ref():
Merged to btr_search_info_update_hash().
btr_cur_t::n_bytes_fields: Replaces n_bytes << 16 | n_fields.
We also remove many redundant checks of btr_search.enabled.
If we are holding any btr_sea::partition::latch, then a nonnull pointer
in buf_block_t::index must imply that the adaptive hash index is enabled.
Reviewed by: Vladislav Lesin
Now that ut_fold_ulint_pair() and ut_fold_binary() are no longer needed for anything else than compatibility with old InnoDB data files that may use innodb_checksum_algorithm=innodb, let us move the code to a single compilation unit. Reviewed by: Vladislav Lesin
btr_cur_t::search_leaf(): Do not attempt to use the adaptive hash index for PAGE_CUR_G or PAGE_CUR_L, because those modes expect an inequal result, and the adaptive hash index can only deliver equal results. btr_cur_t::check_mismatch(): Only handle PAGE_CUR_LE and PAGE_CUR_GE. For PAGE_CUR_LE (bool ge=false), qualify a full match for the last record of a page that is not at the end of the index. Previously, an adaptive hash index lookup would fail when the record is at the end of an index page but not at the end of the index. This would lead to unnecessary rebuild of the adaptive hash index in read-only workloads. Reviewed by: Vladislav Lesin
btr_search_guess_on_hash(): Only set BTR_CUR_HASH_FAIL on actual mismatch. If the page latch cannot be acquired, the hash search might very well have succeeded. Do not count that as a failure, that is, do not unnecessarily invoke btr_search_update_hash_ref() after a normal search. Set cursor->flag=BTR_CUR_HASH_ABORT if the current parameters of the adaptive hash index are not suitable for the search and a call to btr_cur_t::search_info_update() might help. btr_cur_t::search_leaf(): Do not invoke search_info_update() if btr_search_guess_on_hash() failed due to contention. btr_cur_t::pessimistic_search_leaf(): Do not invoke search_info_update() on the change buffer tree. Preivously, this condition was being checked inside search_info_update().
btr_search_info_update_hash(): Do nothing if the record is positioned on the page supremum or infimum pseudo-record. The adaptive hash index can only include user records. This deficiency would cause the adaptive hash index parameters to change between hashing a prefix of 1 field or a prefix of 1 byte. Reviewed by: Vladislav Lesin
btr_search_drop_page_hash_index(): Replace the Boolean parameter with const dict_index_t *not_garbage. If buf_block_t::index points to that, there is no need to acquire btr_sea::partition::latch. The old parameter bool garbage_collect=false is equivalent to the parameter not_garbage=nullptr. The parameter garbage_collect=true will be replaced either with the actual index that is associated with the buffer page, or with a bogus pointer not_garbage=-1 to indicate that any lazily entries for a freed index need to be removed. buf_page_get_low(), buf_page_get_gen(), mtr_t::page_lock(), mtr_t::upgrade_buffer_fix(): Do not invoke btr_search_drop_page_hash_index(). Our caller will have to do it when appropriate. buf_page_create_low(): Keep invoking btr_search_drop_page_hash_index(). This is the normal way of lazily dropping the adaptive hash index after a DDL operation such as DROP INDEX operation. btr_block_get(), btr_root_block_get(), btr_root_adjust_on_import(), btr_read_autoinc_with_fallback(), btr_cur_instant_init_low(), btr_cur_t::search_leaf(), btr_cur_t::pessimistic_search_leaf(), btr_pcur_optimistic_latch_leaves(), dict_stats_analyze_index_below_cur(): Invoke btr_search_drop_page_hash_index(block, index) for pages that may be leaf pages. No adaptive hash index may have been created on anything else than a B-tree leaf page. btr_cur_search_to_nth_level(): Do not invoke btr_search_drop_page_hash_index(), because we are only accessing non-leaf pages and the adaptive hash index may only have been created on leaf pages. btr_page_alloc_for_ibuf() and many other callers of buf_page_get_gen() or similar functions do not invoke btr_search_drop_page_hash_index(), because the adaptive hash index is never created on such pages. If a page in the tablespace was freed as part of a DDL operation and reused for something else, then buf_page_create_low() will take care of dropping the adaptive hash index before the freed page will be modified. It is notable that while the flst_ functions may access pages that are related to allocating B-tree index pages (the BTR_SEG_TOP and BTR_SEG_LEAF linked from the index root page), those pages themselves can never be stored in the adaptive hash index. Therefore, it is not necessary to invoke btr_search_drop_page_hash_index() on them. Reviewed by: Vladislav Lesin
During a workload, an adaptive hash index had been built on UNIQUE INDEX(ID) on SYS_TABLES, and during a DROP TABLE operation the adaptive hash index would be widened to cover also the PRIMARY KEY(NAME) field that the index includes: (ID,NAME). Such an adaptive hash index is unlikely to satisfy (m)any queries. Let us limit the AHI prefix to the unique fields. Reviewed by: Vladislav Lesin
For some reason, page_cur_search_with_match_bytes(), which can speed up append operations (PAGE_CUR_LE used by INSERT), was only enabled if innodb_adaptive_hash_index=ON even though it has nothing to do with the adaptive hash index. Furthermore, mysql/mysql-server@c9bbc83 a.k.a. commit c9bbc83 reduced a limit from 3 to 2 but forgot to adjust the PAGE_N_DIRECTION limit accordingly. We are adjusting that as well. Reviewed by: Vladislav Lesin
page_cur_search_with_match(): Remove rec_get_offsets(), and instead determine the start and end of each field while comparing. page_dir_slot_get_rec(), page_dir_slot_get_rec_validate(): Add a parameter to avoid invoking page_align(). page_cur_dtuple_cmp(): Replaces cmp_dtuple_rec_leaf() for both leaf and non-leaf pages. In SPATIAL INDEX, non-leaf records are special, because the child page number may be part of the comparison. Reviewed by: Vladislav Lesin
|
I had also created the branch bb-10.11-MDEV-35049-rebase, whose current head revision is a253f93. |
Description
This also includes #3574 and #3627 until they have been merged to 10.6.
I suggest to read the individual [commit mesages|https://github.com//pull/3562/commits] (
git log origin/10.6..origin/10.6-MDEV-35190).This is a rewrite of the InnoDB adaptive hash index, as well as a rewrite of record comparisons so that heap memory allocation as well as calls to
rec_get_offsets()will be avoided.MEM_HEAP_BTR_SEARCHandmem_block_info_t::free_blockare removed. We implement a custom memory allocator for the adaptive hash index entries (ahi_node, renamed fromha_node_t). Themem_heapsubsystem is simplfiied.We also make
rec_fold()compute CRC-32C on a contiguous range of bytes, from the start of the record to the last included byte of the last field. Indtuple_fold()we compute the checksum field by field.The function
page_cur_search_with_match_bytes(), which can benefitINSERTinto the end of an index page, will be enabled even wheninnodb_adaptive_hash_index=OFF.btr_search_guess_on_hash()will returnBTR_CUR_HASH_FAILin fewer cases. This will lead to less frequent rebuild of the adaptive hash index and therefore better performance in a read-only workload.Many fields related to comparing records and the adaptive hash index are consolidated into a single
n_bytes_fieldsorleft_bytes_fields. A redundant set of AHI related parameters is removed frombuf_block_t. Thebuf_block_t::ahi_left_bytes_fieldswill useAtomic_relaxed<uint32_t>.Release Notes
The performance of
innodb_adaptive_hash_index=ONwas improved at high concurrency.How can this PR be tested?
I tested this with and without
cmake -DWITH_INNODB_AHI=OFFas well as withcmake -DWITH_INNODB_ROOT_GUESS=OFF.With the
innodb-hashtest.shthat @montywi attached to MDEV-35049 I no longer see any waits for thebtr_sea::partition::latch, and there are no calls tobtr_sea::partition::insertduring the workload. This means that the adaptive hash index remains constant during a constant read-only workload where the data fits in the buffer pool.This needs to be tested with a larger workload that originally reproduced the scalability issue.
Basing the PR against the correct MariaDB version
mainbranch.Part of this could be applicable to 10.5, but the patch is currently is based on the 10.6 branch, because it depends on many performance improvements that are only available starting with 10.6.
PR quality check