MDEV-21322: report slave progress to the master by JackSlateur · Pull Request #1427 · MariaDB/server

JackSlateur · 2019-12-15T15:02:49Z

We reuse the semisync replication mode to get and report
slave progress to the master, where it can then be used via
SHOW SLAVE HOSTS for instance.

Today, we have:

MariaDB [(none)]> SHOW SLAVE HOSTS;
+-----------+-------------------------+------+-----------+
| Server_id | Host                    | Port | Master_id |
+-----------+-------------------------+------+-----------+
|        20 | 2a0a:1580::37 | 3306 |        10 |
|        30 | 10.3.95.72 | 3306 |        10 |
+-----------+-------------------------+------+-----------+
1 row in set (0.000 sec)

With the following master configuration:

rpl_semi_sync_master_enabled = ON

And slaves configuration (with report_host being set by the default):

rpl_semi_sync_slave_enabled = ON

We get:

MariaDB [(none)]> SHOW SLAVE HOSTS;
+-----------+------------------+------+-----------+----------------+----------+--------------------------+
| Server_id | Host             | Port | Master_id | File           | Position | Gtid                     |
+-----------+------------------+------+-----------+----------------+----------+--------------------------+
|     12345 | abweb2.odiso.net | 3306 |        10 | log-bin.000036 | 64937269 | 0-10-246940,10-10-947491 |
|      1234 | abweb3.odiso.net | 3306 |        10 | log-bin.000036 | 64937269 | 0-10-246940,10-10-947491 |
+-----------+------------------+------+-----------+----------------+----------+--------------------------+
2 rows in set (0.155 sec)

In the same setup, a slave without semi_sync enabled simply does not reports log_file, position nor gtid

Please tell me if this is the right way to do

Signed-off-by: Alexandre Bruyelles git@jack.fr.eu.org

JackSlateur · 2020-01-09T07:40:17Z

@andrelkin What do you think ?
On second thought, maybe I should have made two separate MR : one to allow semi-sync to be "dummy", and one to alter the SHOW SLAVE HOSTS command

Tell me if you want

CLAassistant · 2020-03-26T06:29:10Z

All committers have signed the CLA.

grooverdan · 2020-08-21T10:15:07Z

Rather than a new system variable, how does extending the existing to 'rpl_semi_sync_master_wait_point=NEVER' (or "disabled'?) sound? After all its pretty meanless to have a sync wait point variable implies a wait, and doesn't.

andrelkin · 2020-08-24T18:47:29Z

Rather than a new system variable, how does extending the existing to 'rpl_semi_sync_master_wait_point=NEVER' (or "disabled'?) sound? After all its pretty meanless to have a sync wait point variable implies a wait, and doesn't.

I think it's not a bad idea. Thanks @grooverdan! @JackSlateur - could you please consider that?

JackSlateur · 2020-08-24T19:25:22Z

Rather than a new system variable, how does extending the existing to 'rpl_semi_sync_master_wait_point=NEVER' (or "disabled'?) sound? After all its pretty meanless to have a sync wait point variable implies a wait, and doesn't.

I think it's not a bad idea. Thanks @grooverdan! @JackSlateur - could you please consider that?

That is actually a neat idea
I dropped the code from the merge request - I'll rework the code and submit another MR later

robertbindar

Hi @JackSlateur! Thanks for the contribution! I know it's been a long time since you worked on this, so you might not even remember what the feature was about.
Would you be willing to continue the work on this? I can definitely stick around and help until the patch gets merged.
The idea is perfectly fine, implementation the same.
What I think what the patch still needs is:

a NONE extra value for rpl_semi_sync_master_wait_point system var, I don't see in your diff a way to disable waiting on semisync replica (move to dummy mode as you named it :-) ) and where it is checked to see reporting doesn't happen when "dummy" mode is no enabled.
a test that actually shows the reporting works, as far as I can see the test updates you performed only prove that SHOW SLAVE HOSTS is capable of displaying the report data, but don't prove that the result of the SQL command actually displays correct data.

Please let me know if you're still willing to continue the implementation of this, I'd love to guide you to a polished merged feature. Otherwise let me know here and it becomes MariaDB developers job to finish this work.

We reuse the semisync replication mode to get and report replica progress to the master, where it can then be used via SHOW REPLICA HOSTS for instance.

bnestere · 2022-06-09T18:20:43Z

+Rpl_semi_sync_master_clients	0
+show status like 'Rpl_semi_sync_master_status';
+Variable_name	Value
+Rpl_semi_sync_master_status	ON


@grooverdan astutely noticed here that Rpl_semi_sync_master_status is ON when it should be OFF. More generally, should Rpl_semi_sync_master_status ever be ON when Rpl_semi_sync_master_wait_point=NONE? It seems more consistent that the status should always be OFF if the master will never actually wait for an ack. See the following comment for Repl_semi_sync_master::switch_off():

/* Indicate that semi-sync replication is OFF now. * * What should we do when it is disabled? The problem is that we want * the semi-sync replication enabled again when the slave catches up * later. But, it is not that easy to detect that the slave has caught * up. This is caused by the fact that MySQL's replication protocol is * asynchronous, meaning that if the master does not use the semi-sync * protocol, the slave would not send anything to the master. * Still, if the master is sending (N+1)-th event, we assume that it is * an indicator that the slave has received N-th event and earlier ones. * * If semi-sync is disabled, all transactions still update the wait * position with the last position in binlog. But no transactions will * wait for confirmations and the active transaction list would not be * maintained. In binlog dump thread, update_sync_header() checks whether * the current sending event catches up with last wait position. If it * does match, semi-sync will be switched on again. */ void Repl_semi_sync_master::switch_off() { ... }

So my recommendation would be to update the return condition of is_on() to be return m_state && wait_point() != SEMI_SYNC_MASTER_WAIT_POINT_NONE;. Also, Rpl_semi_sync_master_status would then need to be updated to be is_on() rather than m_state in set_export_stats().

bnestere · 2022-06-09T18:23:37Z

+    protocol->store(si->log_file, safe_strlen(si->log_file), &my_charset_bin);
+    protocol->store(si->log_pos);
+    protocol->store(gtid);
+    repl_semisync_master.store_status(protocol);


I think it would make more sense here to use rpl_semi_sync_master_status to avoid double-locking as well as the additional dependency additions.

bnestere · 2022-06-13T14:24:57Z

+SET GLOBAL rpl_semi_sync_master_enabled = OFF;
+DROP TABLE t1;
+SHOW SLAVE HOSTS;
+


I think the test should additionally account for the following cases:

That the reported position of the slave (by SHOW REPLICA HOSTS) actually matches the slave’s gtid_slave_pos value

That the new columns reported by SHOW REPLICA HOSTS works/is correct when using all rpl_semi_sync_master_wait_point options

What does the output of SHOW REPLICA HOSTS show if a slave has semi-sync disabled?

What is the output of SHOW REPLICA HOSTS when a slave is connected but the master has no events in the binlog? (the initial state).

JackSlateur · 2022-07-26T18:19:39Z

@bnestere Hello,
Sadly, I have no longer the required time to work on mariadb
Please feel free to take over the work or close this merge request

Best regards,

+ We are trying to extend the command `SHOW REPLICA HOSTS` that is executed on master, with columns `GTID_state_sent` and `GTID_state_ack`. In order to achieve that we have to extend `thd->slave_info` struct with 2 new entries. This struct needs to be populated and must be accessable to primary. + The first column `GTID_state_sent`: - It is populated by the last event that primary has sent to replica (only for semi-sync slave connection, but without need to know that replica actually obtained request, like it is case in asynchronous replication). - Primary executes `mysql_binlog` that first starts the `binlog_dump` thread. That thread starts `ack_receiver` thread, that returns second column of interest (in first iteration we don't care about). After `binlog_dump` thread creating, file is sent with `send_one_binlog_file()` function. This function is sending event with `send_events()` function, during which `send_event_to_slave()` function is executed. On that place we are creating the `thd->slave_info->gtid_state_sent` struct. + The second column `GTID_state_ack`: - It is populated by the `reply_packet_binlog` (a.k.a. `ack`) that is called, only for semi-sync replication. It is called on 2 places: a) during `binlog_dump` creation, after thread is added to `ack_reciver` and after thread is added as semi-sync slave. This should be first ack received from replica to primary. b) constantly in running phase of `ack_receiver` thread `run()`. - A the end of function `reply_packet_binlog` we are creating the struct `GTID_state_ack` + The third column `Sync_Status` - It should be filled with values `asynchronous`,`semi-sync active`, `semi-sync stale` in order to express the state of replica. - `semi-sync [active/stale]` related states depend on the value of `need_sync` that is updated on master and will be `semi-sync active` iff: - primary and replicas are configured to be in semi-sync mode and - if the event is a transaction's ending event. Otherwise state is `semi-sync stale`. - state is `asynchronous` if semi-sync is disabled on master as fast check and if there thd is not semi-sync thread. + Handling of `rpl_semi_sync_master_timeout= 0` means that primary status `rpl_semi_sync_master_status` will remain ON instead of switching off to async and incrementing the counter updates. + PR closes MariaDB#2374, MariaDB#1427 Reviewer:<brandon.nesterenko@mariadb.com>

LinuxJedi · 2023-07-26T15:58:36Z

Development of this is being continued in PR #2374 based on the work here. I will therefore close this pull request for now. Many thanks for your work on this so far.

+ We are trying to extend the command `SHOW REPLICA HOSTS` that is executed on master, with columns `GTID_state_sent` and `GTID_state_ack`. In order to achieve that we have to extend `thd->slave_info` struct with 2 new entries. This struct needs to be populated and must be accessable to primary. + The first column `GTID_state_sent`: - It is populated by the last event that primary has sent to replica (only for semi-sync slave connection, but without need to know that replica actually obtained request, like it is case in asynchronous replication). - Primary executes `mysql_binlog` that first starts the `binlog_dump` thread. That thread starts `ack_receiver` thread, that returns second column of interest (in first iteration we don't care about). After `binlog_dump` thread creating, file is sent with `send_one_binlog_file()` function. This function is sending event with `send_events()` function, during which `send_event_to_slave()` function is executed. On that place we are creating the `thd->slave_info->gtid_state_sent` struct. + The second column `GTID_state_ack`: - It is populated by the `reply_packet_binlog` (a.k.a. `ack`) that is called, only for semi-sync replication. It is called on 2 places: a) during `binlog_dump` creation, after thread is added to `ack_reciver` and after thread is added as semi-sync slave. This should be first ack received from replica to primary. b) constantly in running phase of `ack_receiver` thread `run()`. - A the end of function `reply_packet_binlog` we are creating the struct `GTID_state_ack` + The third column `Sync_Status` - It should be filled with values `asynchronous`,`semi-sync active`, `semi-sync stale` in order to express the state of replica. - `semi-sync [active/stale]` related states depend on the value of `need_sync` that is updated on master and will be `semi-sync active` iff: - primary and replicas are configured to be in semi-sync mode and - if the event is a transaction's ending event. Otherwise state is `semi-sync stale`. - state is `asynchronous` if semi-sync is disabled on master as fast check and if there thd is not semi-sync thread. + Handling of `rpl_semi_sync_master_timeout= 0` means that primary status `rpl_semi_sync_master_status` will remain ON instead of switching off to async and incrementing the counter updates. + PR closes MariaDB#2374, MariaDB#1427 Reviewer:<brandon.nesterenko@mariadb.com>

+ We are trying to extend the command `SHOW REPLICA HOSTS` that is executed on master, with columns `GTID_state_sent` and `GTID_state_ack`. In order to achieve that we have to extend `thd->slave_info` struct with 2 new entries. This struct needs to be populated and must be accessable to primary. + The first column `GTID_state_sent`: - It is populated by the last event that primary has sent to replica (only for semi-sync slave connection, but without need to know that replica actually obtained request, like it is case in asynchronous replication). - Primary executes `mysql_binlog` that first starts the `binlog_dump` thread. That thread starts `ack_receiver` thread, that returns second column of interest (in first iteration we don't care about). After `binlog_dump` thread creating, file is sent with `send_one_binlog_file()` function. This function is sending event with `send_events()` function, during which `send_event_to_slave()` function is executed. On that place we are creating the `thd->slave_info->gtid_state_sent` struct. + The second column `GTID_state_ack`: - It is populated by the `reply_packet_binlog` (a.k.a. `ack`) that is called, only for semi-sync replication. It is called on 2 places: a) during `binlog_dump` creation, after thread is added to `ack_reciver` and after thread is added as semi-sync slave. This should be first ack received from replica to primary. b) constantly in running phase of `ack_receiver` thread `run()`. - A the end of function `reply_packet_binlog` we are creating the struct `GTID_state_ack` + The third column `Sync_Status` - It should be filled with values `asynchronous`,`semi-sync active`, `semi-sync stale` in order to express the state of replica. - `semi-sync [active/stale]` related states depend on the value of `need_sync` that is updated on master and will be `semi-sync active` iff: - primary and replicas are configured to be in semi-sync mode and - if the event is a transaction's ending event. Otherwise state is `semi-sync stale`. - state is `asynchronous` if semi-sync is disabled on master as fast check and if there thd is not semi-sync thread. + Handling of `rpl_semi_sync_master_timeout= 0` means that primary status `rpl_semi_sync_master_status` will remain ON instead of switching off to async and incrementing the counter updates. + PR closes #2374, #1427 Reviewer:<brandon.nesterenko@mariadb.com> Thanks <knielsen@knielsen-hq.org> for reviewing the failure and suppresions.

+ We are trying to extend the command `SHOW REPLICA HOSTS` that is executed on master, with columns `GTID_state_sent` and `GTID_state_ack`. In order to achieve that we have to extend `thd->slave_info` struct with 2 new entries. This struct needs to be populated and must be accessable to primary. + The first column `GTID_state_sent`: - It is populated by the last event that primary has sent to replica (only for semi-sync slave connection, but without need to know that replica actually obtained request, like it is case in asynchronous replication). - Primary executes `mysql_binlog` that first starts the `binlog_dump` thread. That thread starts `ack_receiver` thread, that returns second column of interest (in first iteration we don't care about). After `binlog_dump` thread creating, file is sent with `send_one_binlog_file()` function. This function is sending event with `send_events()` function, during which `send_event_to_slave()` function is executed. On that place we are creating the `thd->slave_info->gtid_state_sent` struct. + The second column `GTID_state_ack`: - It is populated by the `reply_packet_binlog` (a.k.a. `ack`) that is called, only for semi-sync replication. It is called on 2 places: a) during `binlog_dump` creation, after thread is added to `ack_reciver` and after thread is added as semi-sync slave. This should be first ack received from replica to primary. b) constantly in running phase of `ack_receiver` thread `run()`. - A the end of function `reply_packet_binlog` we are creating the struct `GTID_state_ack` + The third column `Sync_Status` - It should be filled with values `asynchronous`,`semi-sync active`, `semi-sync stale` in order to express the state of replica. - `semi-sync [active/stale]` related states depend on the value of `need_sync` that is updated on master and will be `semi-sync active` iff: - primary and replicas are configured to be in semi-sync mode and - if the event is a transaction's ending event. Otherwise state is `semi-sync stale`. - state is `asynchronous` if semi-sync is disabled on master as fast check and if there thd is not semi-sync thread. + Handling of `rpl_semi_sync_master_timeout= 0` means that primary status `rpl_semi_sync_master_status` will remain ON instead of switching off to async and incrementing the counter updates. + PR closes MariaDB#2374, MariaDB#1427 Reviewer:<brandon.nesterenko@mariadb.com>

JackSlateur force-pushed the slave_report branch 3 times, most recently from 6b39028 to fa036e3 Compare December 15, 2019 19:41

an3l added the license-mca label Dec 16, 2019

an3l added this to the 10.5 milestone Dec 16, 2019

an3l changed the title ~~repl: report slave progress to the master~~ MDEV-21322: report slave progress to the master Dec 16, 2019

an3l assigned andrelkin Dec 16, 2019

an3l requested a review from andrelkin December 16, 2019 06:49

JackSlateur force-pushed the slave_report branch 10 times, most recently from ce7a279 to 330af02 Compare August 18, 2020 19:20

JackSlateur force-pushed the slave_report branch 3 times, most recently from 116082f to 0e27fde Compare August 22, 2020 12:07

robertbindar self-assigned this Dec 2, 2021

robertbindar suggested changes Dec 9, 2021

View reviewed changes

JackSlateur requested a review from ottok as a code owner December 10, 2021 16:40

JackSlateur force-pushed the slave_report branch from 48ce40e to 0e27fde Compare December 10, 2021 16:41

JackSlateur force-pushed the slave_report branch 2 times, most recently from b03df06 to 38d0b33 Compare February 1, 2022 19:50

JackSlateur requested a review from robertbindar February 1, 2022 20:33

grooverdan assigned grooverdan and unassigned robertbindar and andrelkin Jun 7, 2022

grooverdan force-pushed the slave_report branch from 38d0b33 to b7835cb Compare June 8, 2022 08:53

grooverdan changed the base branch from 10.8 to 10.10 June 8, 2022 08:53

MDEV-21322: report replica progress to the master

1f4aa82

We reuse the semisync replication mode to get and report replica progress to the master, where it can then be used via SHOW REPLICA HOSTS for instance.

grooverdan force-pushed the slave_report branch from b7835cb to 1f4aa82 Compare June 9, 2022 04:31

bnestere reviewed Jun 9, 2022

View reviewed changes

bnestere reviewed Jun 13, 2022

View reviewed changes

an3l assigned an3l and unassigned grooverdan Dec 2, 2022

an3l removed their assignment Mar 28, 2023

LinuxJedi closed this Jul 26, 2023

Navigation Menu

Uh oh!

Uh oh!

MDEV-21322: report slave progress to the master - #1427

MDEV-21322: report slave progress to the master#1427
JackSlateur wants to merge 1 commit into
MariaDB:10.10from
JackSlateur:slave_report

JackSlateur commented Dec 15, 2019 •

edited

Loading

Uh oh!

JackSlateur commented Jan 9, 2020

Uh oh!

CLAassistant commented Mar 26, 2020 •

edited

Loading

Uh oh!

grooverdan commented Aug 21, 2020

Uh oh!

andrelkin commented Aug 24, 2020

Uh oh!

JackSlateur commented Aug 24, 2020

Uh oh!

robertbindar left a comment

Uh oh!

bnestere Jun 9, 2022 •

edited

Loading

Uh oh!

bnestere Jun 9, 2022

Uh oh!

bnestere Jun 13, 2022

Uh oh!

JackSlateur commented Jul 26, 2022

Uh oh!

LinuxJedi commented Jul 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

8 participants

Uh oh!

Uh oh!

Conversation

JackSlateur commented Dec 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JackSlateur commented Jan 9, 2020

Uh oh!

CLAassistant commented Mar 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grooverdan commented Aug 21, 2020

Uh oh!

andrelkin commented Aug 24, 2020

Uh oh!

JackSlateur commented Aug 24, 2020

Uh oh!

robertbindar left a comment

Choose a reason for hiding this comment

Uh oh!

bnestere Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnestere Jun 9, 2022

Choose a reason for hiding this comment

Uh oh!

bnestere Jun 13, 2022

Choose a reason for hiding this comment

Uh oh!

JackSlateur commented Jul 26, 2022

Uh oh!

LinuxJedi commented Jul 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

8 participants

JackSlateur commented Dec 15, 2019 •

edited

Loading

CLAassistant commented Mar 26, 2020 •

edited

Loading

bnestere Jun 9, 2022 •

edited

Loading