From 5367c9f92115e4343369e0167eed80bae26c178a Mon Sep 17 00:00:00 2001 From: Arvin Date: Tue, 12 Feb 2019 10:26:46 +0800 Subject: [PATCH 1/4] init bypass-journal-bp --- site/bps/BP-bypass-journal-ledger.md | 97 ++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 site/bps/BP-bypass-journal-ledger.md diff --git a/site/bps/BP-bypass-journal-ledger.md b/site/bps/BP-bypass-journal-ledger.md new file mode 100644 index 00000000000..0b44076e15e --- /dev/null +++ b/site/bps/BP-bypass-journal-ledger.md @@ -0,0 +1,97 @@ +--- +title: "BP-38: bypass journal ledger" +issue: https://github.com/apache/bookkeeper/issues/1918 +state: "Under Discussion" +release: "N/A" +--- + +### Motivation + +To guarantee high durability, BK write journal before flush data to persistent device which will cause two write of data. +At the presence of replicating and auto-recovery mechanism, the two-write is a bit waste of the persistent device bandwidth, +especially on the [scenarios](https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-14+Relax+durability) which prefer weak durability guarantee. +This proposal is aimed at providing bypass journal ledger, this feature includes these parts work: + - add new write flag `BYPASS_JOURNAL` to existing protocol + - impl the newly write flag at the client side and server side + +The details of changes are listed in Section “[Proposed Changes](#proposed-changes)”. + +### Public Interfaces + +This feature will introduce new 'WRITE_FLAG' to enable single write storage impl. +Like `DEFERRED_SYNC`, when we construct `WriteHandle`, we can pass `BYPASS_JOURNAL`, then every `AddRequest` will carry this flag. +As for monitoring, we can add metrics to record the main time interval during the io path. +The detailed compatibility related stuff is in Section “[CDMP](#compatibility-deprecation-and-migration-plan)”. + +### Proposed Changes + +While the main approach is based on [`WRITE_FLAG` impl](https://github.com/apache/bookkeeper/pull/742), +the actual implementation is more tricky. In my opinion, there are three different solution: + +1. Relax LAC protocol + + Just modify server side code, if the write flag is `BYPASS_JOURNAL`, after write to `LegerStorage`(the data maybe in the memTable, or the buffer of File, or the os cache), +return to the client directly. +This impl is like [disable syncData](https://github.com/apache/bookkeeper/issues/753), once all the replica fails, the BK cluster can't recovery from it. +Note, the user shall know the weak durability for his use case when using this impl. + +2. Extend `Force` API + + Like `DEFERRED_SYNC`, the normal 'bypass-journal' operation don't advance LAC on the client. +Only advance LAC using `force` api. To support this new 'bypass-journal' option, we need theses changes: + + - Add persistent callback to LedgerStorage + + Maintain non-persistent entry list and `maxPersistentEntryId` on `LedgerDescriptor` +If the entry is 'bypass-journal', the callback for this entry is not null, and it will receives notification after persistent. + + - Extend the force request + + If the force request contains bypass journal option, get `maxPersistentEntryId` from `LedgerDescriptor` + or impl force method on `LedgerDescriptor`, which force the entry to persistent device through `LedgerStorage`. + + In addition to these changes, we should consider who is responsible for the `force` execution? How often to schedule 'force'? + +3. Add NonPersistentLAC semantic to existing LAC protocol + + To not violate LAC semantics like method 1, we can add `NonPersistentLAC` semantic, this includes these changes: + + - Add persistent callback to LedgerStorage + + Maintain non-persistent entry list and `maxPersistentEntryId` on `LedgerDescriptor`. + + - Wire protocol changes + + Add optional `maxPersistentEntryId` field to `AddResponse`. When the `AddRequest` has 'bypass-journal' option, the server get `maxPersistentEntryId` from `LedgerDescriptor`. + + - Client side changes + + Add `nonPersistentLAC` to WriteHandle. WriteHandle with 'bypass-journal' option updates the LAC using `maxPersistentEntryId`, and update `nonPersistentLAC` if receives enough ack. + For normal WriteHandle, the `nonPersistentLAC` is set to null to keep original LAC advance logic. + +### Compatibility, Deprecation, and Migration Plan + +As this new feature only add one ledger creating option and corresponding ledger implementation. It has no effect +on existing users. After this feature is implemented, user can use this function directly on new released version which includes this implementation. +When user use the latest client which has the `BYPASS_JOURNAL` write flag, but the bookie is old versions which can't do bypass-journal process, + the bookie will reject this type write. Old client has no chance to use this new write flag, and the new bookie can handle as usual. + +### Test Plan + +- unit tests for newly introduced write_flag at client side and server side +- end-to-end tests for the new Protocol request/response +- compat tests (client with this new writeflags will not be able to write on old bookies) + +### Rejected Alternatives + +To avoid two write of data, we can write the data to one log abstraction in the bookie, either by bypassing journal or + bypassing entryLog. If we choose bypassing entryLog, then we will do lots of work to enable rich read on journal, + which is more huge work compared with bypassing journal. + +### sub-task/ small basic component task + +- [ ] server side changes (bypass-journal) +- [ ] client side implementation (writeflag BYPASS_JOURNAL implementation on the client side) +- [ ] compat tests (client with writeflags will not be able to write on old bookies) +- [ ] docs update + From 53fa1ef3217b74ed9ac0774c37f569ccfc5900e6 Mon Sep 17 00:00:00 2001 From: Arvin Date: Tue, 12 Feb 2019 10:59:30 +0800 Subject: [PATCH 2/4] regulate bp-38 --- ...edger.md => BP-38-bypass-journal-ledger.md} | 18 ++++++------------ site/community/bookkeeper_proposals.md | 3 ++- 2 files changed, 8 insertions(+), 13 deletions(-) rename site/bps/{BP-bypass-journal-ledger.md => BP-38-bypass-journal-ledger.md} (85%) diff --git a/site/bps/BP-bypass-journal-ledger.md b/site/bps/BP-38-bypass-journal-ledger.md similarity index 85% rename from site/bps/BP-bypass-journal-ledger.md rename to site/bps/BP-38-bypass-journal-ledger.md index 0b44076e15e..c159293f2a3 100644 --- a/site/bps/BP-bypass-journal-ledger.md +++ b/site/bps/BP-38-bypass-journal-ledger.md @@ -30,8 +30,8 @@ the actual implementation is more tricky. In my opinion, there are three differe 1. Relax LAC protocol - Just modify server side code, if the write flag is `BYPASS_JOURNAL`, after write to `LegerStorage`(the data maybe in the memTable, or the buffer of File, or the os cache), -return to the client directly. + Modify server side code mostly and don't change legerHandle's LAC advance logic, if the write flag is `BYPASS_JOURNAL`, after write to `LegerStorage`(the data maybe in the memTable, or the buffer of File, or the os cache), +bookie return result to the client directly. This impl is like [disable syncData](https://github.com/apache/bookkeeper/issues/753), once all the replica fails, the BK cluster can't recovery from it. Note, the user shall know the weak durability for his use case when using this impl. @@ -42,8 +42,8 @@ Only advance LAC using `force` api. To support this new 'bypass-journal' option, - Add persistent callback to LedgerStorage - Maintain non-persistent entry list and `maxPersistentEntryId` on `LedgerDescriptor` -If the entry is 'bypass-journal', the callback for this entry is not null, and it will receives notification after persistent. + Maintain non-persistent entry list(necessary for out of order entry arriving) and `maxPersistentEntryId` on `LedgerDescriptor` +If the entry is 'bypass-journal', the callback for this entry is not null, and it will receives notification after persistent through legerStorage. - Extend the force request @@ -62,12 +62,12 @@ If the entry is 'bypass-journal', the callback for this entry is not null, and i - Wire protocol changes - Add optional `maxPersistentEntryId` field to `AddResponse`. When the `AddRequest` has 'bypass-journal' option, the server get `maxPersistentEntryId` from `LedgerDescriptor`. + Add optional `maxPersistentEntryId` field to `AddResponse`. When the `AddRequest` has 'bypass-journal' option, the server get `maxPersistentEntryId` from `LedgerDescriptor` to construct response. - Client side changes Add `nonPersistentLAC` to WriteHandle. WriteHandle with 'bypass-journal' option updates the LAC using `maxPersistentEntryId`, and update `nonPersistentLAC` if receives enough ack. - For normal WriteHandle, the `nonPersistentLAC` is set to null to keep original LAC advance logic. + For normal WriteHandle, the `nonPersistentLAC` is set to null to indicate to keep original LAC advance logic. ### Compatibility, Deprecation, and Migration Plan @@ -87,11 +87,5 @@ When user use the latest client which has the `BYPASS_JOURNAL` write flag, but t To avoid two write of data, we can write the data to one log abstraction in the bookie, either by bypassing journal or bypassing entryLog. If we choose bypassing entryLog, then we will do lots of work to enable rich read on journal, which is more huge work compared with bypassing journal. - -### sub-task/ small basic component task -- [ ] server side changes (bypass-journal) -- [ ] client side implementation (writeflag BYPASS_JOURNAL implementation on the client side) -- [ ] compat tests (client with writeflags will not be able to write on old bookies) -- [ ] docs update diff --git a/site/community/bookkeeper_proposals.md b/site/community/bookkeeper_proposals.md index c15e1d26d5c..1b4ef82a297 100644 --- a/site/community/bookkeeper_proposals.md +++ b/site/community/bookkeeper_proposals.md @@ -85,7 +85,7 @@ using Google Doc. This section lists all the _bookkeeper proposals_ made to BookKeeper. -*Next Proposal Number: 38* +*Next Proposal Number: 39* ### Inprogress @@ -109,6 +109,7 @@ Proposal | State [BP-35: 128 bits support](../../bps/BP-35-128-bits-support) | Accepted [BP-36: Stats documentation annotation](../../bps/BP-36-stats-documentation-annotation) | Accepted [BP-37: Improve configuration management for better documentation](../../bps/BP-37-conf-documentation) | Accepted +[BP-38: Bypass journal ledger](../../bps/BP-38-bypass-journal-ledger) | Draft ### Adopted From 0f687c7c128cbe7a9633e7468bcd75201a82a012 Mon Sep 17 00:00:00 2001 From: Arvin Date: Tue, 12 Feb 2019 15:28:51 +0800 Subject: [PATCH 3/4] improve motivation description --- site/bps/BP-38-bypass-journal-ledger.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site/bps/BP-38-bypass-journal-ledger.md b/site/bps/BP-38-bypass-journal-ledger.md index c159293f2a3..3a9e777b659 100644 --- a/site/bps/BP-38-bypass-journal-ledger.md +++ b/site/bps/BP-38-bypass-journal-ledger.md @@ -8,8 +8,7 @@ release: "N/A" ### Motivation To guarantee high durability, BK write journal before flush data to persistent device which will cause two write of data. -At the presence of replicating and auto-recovery mechanism, the two-write is a bit waste of the persistent device bandwidth, -especially on the [scenarios](https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-14+Relax+durability) which prefer weak durability guarantee. +But we may not need this level of persistence under [scenarios](https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-14+Relax+durability) which prefer weak durability guarantee. This proposal is aimed at providing bypass journal ledger, this feature includes these parts work: - add new write flag `BYPASS_JOURNAL` to existing protocol - impl the newly write flag at the client side and server side From 611ed44ab4b47e1048b6e6be49634d2cfffabc67 Mon Sep 17 00:00:00 2001 From: Arvin Date: Fri, 15 Feb 2019 21:42:42 +0800 Subject: [PATCH 4/4] update bp38 --- site/bps/BP-38-bypass-journal-ledger.md | 35 ++++++++++++++++++------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/site/bps/BP-38-bypass-journal-ledger.md b/site/bps/BP-38-bypass-journal-ledger.md index 3a9e777b659..1f9cf686845 100644 --- a/site/bps/BP-38-bypass-journal-ledger.md +++ b/site/bps/BP-38-bypass-journal-ledger.md @@ -12,7 +12,22 @@ But we may not need this level of persistence under [scenarios](https://cwiki.ap This proposal is aimed at providing bypass journal ledger, this feature includes these parts work: - add new write flag `BYPASS_JOURNAL` to existing protocol - impl the newly write flag at the client side and server side - + +The usage will looks like this: +```java +WriteHandle writeHandle = newCreateLedgerOp() + .withEnsembleSize(3) + .withWriteQuorumSize(3) + .withAckQuorumSize(3) + .withPassword(PASSWORD) + .withWriteFlags(WriteFlag.BYPASS_JOURNAL) + .execute().get(); +// append data as usual, but the lac won't advance +writeHandle.appendAsync(...); +// force to advance lac +writeHandle.force(); +writeHandle.closeAsync(); +``` The details of changes are listed in Section “[Proposed Changes](#proposed-changes)”. ### Public Interfaces @@ -27,7 +42,7 @@ The detailed compatibility related stuff is in Section “[CDMP](#compatibility- While the main approach is based on [`WRITE_FLAG` impl](https://github.com/apache/bookkeeper/pull/742), the actual implementation is more tricky. In my opinion, there are three different solution: -1. Relax LAC protocol +1. Relax LAC protocol(rejected due to violate LAC protocol) Modify server side code mostly and don't change legerHandle's LAC advance logic, if the write flag is `BYPASS_JOURNAL`, after write to `LegerStorage`(the data maybe in the memTable, or the buffer of File, or the os cache), bookie return result to the client directly. @@ -39,19 +54,19 @@ Note, the user shall know the weak durability for his use case when using this i Like `DEFERRED_SYNC`, the normal 'bypass-journal' operation don't advance LAC on the client. Only advance LAC using `force` api. To support this new 'bypass-journal' option, we need theses changes: - - Add persistent callback to LedgerStorage + - Add force interface to `LedgerDescriptor` and LedgerStorage - Maintain non-persistent entry list(necessary for out of order entry arriving) and `maxPersistentEntryId` on `LedgerDescriptor` -If the entry is 'bypass-journal', the callback for this entry is not null, and it will receives notification after persistent through legerStorage. + Used to flush the data on memTable to persistent device for specific ledger, and this make senses with "one entry logger per ledger" feature. + For general SortedLedgerStorage, this "force" operation may has some overhead due to flush data too early. - Extend the force request - If the force request contains bypass journal option, get `maxPersistentEntryId` from `LedgerDescriptor` - or impl force method on `LedgerDescriptor`, which force the entry to persistent device through `LedgerStorage`. + If the force request contains bypass journal option, execute force method of `LedgerDescriptor`, which will call LedgerStorage's force interface to flush received entries to persistent device. In addition to these changes, we should consider who is responsible for the `force` execution? How often to schedule 'force'? + To simply the design and give more choices to apps, these choices are up to user. -3. Add NonPersistentLAC semantic to existing LAC protocol +3. Add NonPersistentLAC semantic to existing LAC protocol(rejected due to infeasible and complex) To not violate LAC semantics like method 1, we can add `NonPersistentLAC` semantic, this includes these changes: @@ -68,6 +83,8 @@ If the entry is 'bypass-journal', the callback for this entry is not null, and i Add `nonPersistentLAC` to WriteHandle. WriteHandle with 'bypass-journal' option updates the LAC using `maxPersistentEntryId`, and update `nonPersistentLAC` if receives enough ack. For normal WriteHandle, the `nonPersistentLAC` is set to null to indicate to keep original LAC advance logic. +As we only introduce `BYPASS_JOURNAL` write flag to add request, it's better not change existing fencing logic. So fence operation shall falls into journal io path. + ### Compatibility, Deprecation, and Migration Plan As this new feature only add one ledger creating option and corresponding ledger implementation. It has no effect @@ -77,7 +94,7 @@ When user use the latest client which has the `BYPASS_JOURNAL` write flag, but t ### Test Plan -- unit tests for newly introduced write_flag at client side and server side +- unit tests for newly introduced write_flag and extended force at client side and server side - end-to-end tests for the new Protocol request/response - compat tests (client with this new writeflags will not be able to write on old bookies)