MDEV-38147 Mariadb error 1950 after SST - #5316
Open
ayurchen wants to merge 2 commits into
Open
Conversation
Contributor
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
Contributor
Author
|
/gemini review |
Contributor
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
mariadb-TeemuOllakka
requested review from
mariadb-TeemuOllakka
and removed request for
temeo
July 2, 2026 09:34
ayurchen
force-pushed
the
MDEV-38147-mariadb-error-1950-after-sst
branch
2 times, most recently
from
July 5, 2026 21:46
f5dec0e to
3363fc8
Compare
mariadb-TeemuOllakka
requested changes
Jul 6, 2026
mariadb-TeemuOllakka
left a comment
mariadb-TeemuOllakka
left a comment
Contributor
There was a problem hiding this comment.
Initial comments for issues which caught my eye.
I could not yet fully understand the change in recovery logic, need to re-visit it later on.
ayurchen
force-pushed
the
MDEV-38147-mariadb-error-1950-after-sst
branch
3 times, most recently
from
July 7, 2026 14:53
b4852ec to
8a98a1e
Compare
mariadb-TeemuOllakka
requested changes
Jul 8, 2026
mariadb-TeemuOllakka
left a comment
mariadb-TeemuOllakka
left a comment
Contributor
There was a problem hiding this comment.
@ayurchen Here's the comments for my second review pass.
The review is not complete, I haven't gone through mariabackup changes fully yet.
Contributor
|
The commit message for MDEV-38147 part says:
This is not entirely correct: Due to change introduced in 41b435f (MDEV-33211), the provider |
ayurchen
force-pushed
the
MDEV-38147-mariadb-error-1950-after-sst
branch
2 times, most recently
from
July 29, 2026 08:36
a713b71 to
8674eec
Compare
After a mariabackup SST the joiner could fail with
ER_GTID_STRICT_OUT_OF_ORDER (error 1950)
while re-binlogging transactions received over IST.
The cause is that the binary log copied from the donor carries a
Gtid_list whose position can be ahead of the storage-engine snapshot
because getting snapshot and binlog are no mutually atomic, so transactions
can be present in the copied binlog that are not committed in the copied
engine snapshot.
After the SST the joiner reports the (committed) engine position to the
cluster, IST resends those transactions, and re-binlogging them under
gtid_strict_mode=ON collides with the ahead Gtid_list -> error 1950.
(MDEV-34483 made the engine snapshot stop short of the binlog, which is
what exposed this.)
The copied binary log carries no transactions the joiner needs - only a
Gtid_list - so instead of shipping and then having to truncate/reconcile
it, the joiner now starts a fresh binary log and seeds its GTID position
from the storage-engine checkpoint during recovery. That checkpoint is
the committed cluster position, i.e. exactly where IST resumes, so the
joiner's binary log stays in lockstep with the rest of the cluster and
no out-of-order GTID can occur.
This works for both wsrep_gtid_mode settings; only the binlog domain of
the cluster stream differs:
- wsrep_gtid_mode=ON : wsrep_gtid_domain_id (cluster writes are
re-tagged to it), which is the domain stored in the checkpoint;
- wsrep_gtid_mode=OFF: gtid_domain_id (cluster writes keep the node's
configured domain).
Async-replica positions (mysql.gtid_slave_pos) are part of the engine
snapshot and survive the SST unchanged, so a Galera node can still serve
as an async master or replica across the SST.
This commit:
- sql/log.cc: adds wsrep_seed_binlog_gtid_state(), called from
do_binlog_recovery() when the joiner has no binary log, seeding the
binlog GTID state for the cluster domain to the SE checkpoint position.
- scripts/wsrep_sst_mariabackup.sh: no longer moves the donor's binary
log into place on the joiner.
- extra/mariabackup: backward compatibility: keep shipping binlog file
in SST but
- on donor fix the race between rotation and shipping so that the file
shipped is the one that had been rotated;
- on joiner discard shipped binlog in favour of one generated by
wsrep_seed_binlog_gtid_state().
- sql/wsrep_sst.cc: logs the position actually adopted from storage
(the authoritative post-SST position) rather than the script-reported
one.
- sql/handler.cc: downgrades the "Discovered discontinuity in recovered
wsrep transaction XIDs" message in wsrep_order_and_check_continuity()
from warning to debug level. With parallel appliers a snapshot
routinely captures prepared XIDs that are not contiguous with the
engine checkpoint, so this is normal during SST recovery and of no
value in regular operation; the transactions past the checkpoint are
re-delivered by the cluster (IST/SST) regardless.
- Adds an MDEV-38147 MTR test reproducing the issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With log_bin=ON a transaction is committed via two-phase commit (the binary log is the second participant), so it passes through the InnoDB XA-prepare state. While a donor is held in BLOCK_COMMIT for a mariabackup backup, its parallel appliers (wsrep_slave_threads > 1) leave one or more such writesets prepared-but-not-yet-committed, and the snapshot captures them. On a freshly SST'd joiner nothing resolves these prepared transactions: binlog crash recovery does not run (the joiner has no in-use binlog to recover from), and the wsrep continuity-based commit is inactive because wsrep_emulate_bin_log is FALSE when log_bin is ON. The leftover prepared transactions then abort startup with "Found <N> prepared transactions!". Note this does not depend on the prepared set being non-contiguous - even a contiguous run aborts, because nothing commits or rolls it back. Rollback these transactions in xarecover_handlerton(). If rollback fails flag error to cause unireg_abort().
ayurchen
force-pushed
the
MDEV-38147-mariadb-error-1950-after-sst
branch
from
August 1, 2026 10:23
8674eec to
59ba47d
Compare
mariadb-TeemuOllakka
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request created in: https://jira.mariadb.org/browse/MDEV-38147
Investigation into MDEV-38147 revealed that with --galera-info option given during SST mariabackup rotates the binlog and ships it to joiner. The file is likely to contain wrong Gtid_list info and is used on joiner to initialize binlog.
Since that file is useless, don't rotate the binlog and ship the file, instead the joiner can generate its own correct Gtid_list.
MDEV-40179 - rollback orphaned prepared transactions.