Implement simultaneous open extension by vyzo · Pull Request #42 · multiformats/go-multistream

vyzo · 2019-08-26T11:15:53Z

TBD:

tests!
use parallel goroutines for writing in simultaneous read/write paths

vyzo · 2019-09-21T10:25:37Z

raulk

I did a quick read through this and it looks sane. Nice job! I want to do a more thorough review — this is mildly intricate in terms of possible branching and it’ll be on our critical path to everything. Can we add fuzzing tests?

vyzo · 2019-09-25T08:30:41Z

@raulk what scenarios do you want to test? We can certainly add more tests.

aarshkshah1992 · 2021-01-07T13:27:21Z

+		tok, err = ReadNextToken(rwc)
+
+		if err == io.EOF {
+			return "", ErrNotSupported


@vyzo When can this EOF happen ? I mean, why do we treat this error separately ?

EOF will happen when the remote side gives up because we have no protocols in common. In general, it's best to avoid returning EOF for actual error cases anyways (usually, you want to either return ErrUnexpectedEOF, or a better error like we're doing here).

aarshkshah1992 · 2021-01-12T13:02:06Z

Ping @Stebalien @marten-seemann for review.

Stebalien

The idea is sound. I'd actually go a bit further and break much of this out into helper functions that can be reused with the other functions.

We seem to write and send multiple delimited strings at once quite frequently, that should have a helper.
Some of this code now duplicates existing code.

Stebalien · 2021-01-16T01:44:09Z


+// Performs protocol negotiation with the simultaneous open extension; the returned boolean
+// indicator will be true if we should act as a server.
+func SelectWithSimopen(protos []string, rwc io.ReadWriteCloser) (string, bool, error) {


Nit: SelectProtoOrFailSimultanious, or something like that.

It's unclear that this is "select proto or fail".

We're not paying by the letter.

This is done.

Stebalien · 2021-01-16T01:54:46Z

+}
+
+func simOpen(protos []string, rwc io.ReadWriteCloser) (string, bool, error) {
+	retries := 3


There's really no point in re-trying, or even building that into the protocol. If we collide, our randomness sources are likely identical and we'll never finish.

I agree, the likelihood is abysmal. Have removed this.

Stebalien · 2021-01-16T01:58:03Z

+
+		// this skips pipelined protocol negoatiation
+		// keep reading until the token starts with select:
+		if strings.HasPrefix(tok, "select:") {


We should skip exactly one protocol. Otherwise, if someone names their protocol select:, we'll have problems.

@Stebalien I am not sure what you mean here.

Are you talking about the exact scenario where a peer sends "mutlsitream"|"iamclient"|"select:"|"select:nonce"

where that first "select:" is a security protocol the user wants to negotiate ? I agree this scenario will create problems.

But, is this the ONLY scenario you have in mind ? Does this really warrant fixing ?

Yes, that's the scenario I have in mind. My concern isn't really "fixing a bug" but rather "simplifying the spec". If we go with the current approach, the multistream-select spec would need to say "protocols beginning with select: are reserved" and "iamclient" would become an integral part of the spec. If we simply skip exactly one protocol, we can define "iamclient" to be it's own protocol.

The "iamclient" (maybe /bootstrap/simultaneous?) protocol:

Send exactly one preferred protocol.

Send exactly one select:nonce message to pick the winner.

The winner recursively uses multistream-select to pick the final protocol. NOTE: this would require sending a new multistream header as well.

The beauty of this approach is that there's nothing inherently special about the "iamclient" protocol from a spec standpoint. The implementation can take advantage of the multistream-select ambiguity and pipeline "iamclient" with the "preferred protocol" from step 1, but that's not required. A simpler implementation could just:

Negotiate one of "iamclient" or a set of other security transports (one round-trip per step).

If "iamclient" is selected, send the preferred protocol and a nonce.

Once the winner is picked, the winner negotiates their chosen protocol, potentially using the "preferred" protocol.

This version takes an extra round-trip but can be implemented as two compossible protocols. Later, as the libp2p implementation matures, it can be re-implemented to remove the extra round-trip without changing the spec.

TL;DR: it allows us to write the spec as follows:

Multistream protocol.

"iamclient" protocol for selecting a initiator when a simultaneous connect happens.

Optional: oh, by the way, you can combine these as follows ... to save a round-trip.

@Stebalien This is done.

Stebalien · 2021-01-16T01:58:46Z

+	var mybig, peerbig big.Int
+	var iamserver bool
+	mybig.SetBytes(mynonce)
+	peerbig.SetBytes(peernonce)
+
+	switch mybig.Cmp(&peerbig) {
+	case -1:
+		// peer nonce bigger, he is client
+		iamserver = true
+
+	case 1:
+		// my nonce bigger, i am client
+		iamserver = false
+
+	case 0:


https://golang.org/pkg/bytes/#Compare

This is done.

aarshkshah1992

@Stebalien

Have addresses your review save one comment(#42 (comment)).

Please take a look.

aarshkshah1992 · 2021-01-16T06:35:37Z


+// Performs protocol negotiation with the simultaneous open extension; the returned boolean
+// indicator will be true if we should act as a server.
+func SelectWithSimopen(protos []string, rwc io.ReadWriteCloser) (string, bool, error) {


This is done.

aarshkshah1992 · 2021-01-16T07:00:57Z

+}
+
+func simOpen(protos []string, rwc io.ReadWriteCloser) (string, bool, error) {
+	retries := 3


I agree, the likelihood is abysmal. Have removed this.

aarshkshah1992 · 2021-01-16T07:03:37Z

+	var mybig, peerbig big.Int
+	var iamserver bool
+	mybig.SetBytes(mynonce)
+	peerbig.SetBytes(peernonce)
+
+	switch mybig.Cmp(&peerbig) {
+	case -1:
+		// peer nonce bigger, he is client
+		iamserver = true
+
+	case 1:
+		// my nonce bigger, i am client
+		iamserver = false
+
+	case 0:


This is done.

aarshkshah1992 · 2021-01-16T07:34:31Z

+
+		// this skips pipelined protocol negoatiation
+		// keep reading until the token starts with select:
+		if strings.HasPrefix(tok, "select:") {


@Stebalien I am not sure what you mean here.

Are you talking about the exact scenario where a peer sends "mutlsitream"|"iamclient"|"select:"|"select:nonce"

where that first "select:" is a security protocol the user wants to negotiate ? I agree this scenario will create problems.

But, is this the ONLY scenario you have in mind ? Does this really warrant fixing ?

aarshkshah1992 · 2021-01-21T06:10:23Z

@Stebalien Have addressed all your comments. Please can you take a look ?

marten-seemann

Some nits here, but one more serious point: I really think we should reduce the size of the random value we send. A collision probability of 2^-32 should be good enough here, and would simplify things quite a bit.

marten-seemann · 2021-01-22T02:14:42Z

+	}
+
+	var iamserver bool
+	switch bytes.Compare(mynonce, peernonce) {


Will this yield the correct result if one peer sends a "2" and the other one a "0000001"?

On a more general note (and as I've argued on the specs PR), I still believe that using a 256-bit number here is excessive. We definitely can live with a collision probability of 2^-64 (for that matter, 2^-32 would be totally fine as well, and might simplify stuff in JS, where 64 bit numbers tend to be more problematic), which would allow us to use a simple uint64 / uint32 comparison here.

why? It doesn't cost anything to use bigger nonces, which makes the probability of collision astronomically low.

Why? Because it simplifies the code. I'm still not sure if bytes.Compare will give the correct result if the number is encoded with leading 0s.

With a 32 bit nonce the chance is already astronomically small. At this probability, you'd expect a collision every 2^31 connection attempts. Assuming you're performing 10 simultaneous connects per second (I don't think any node will ever do this), you'll have to wait for almost 7 years until you get the first collision. With a 64 bit nonce, you'd have to wait about 30 billion years, twice the age of the universe.

@vyzo @marten-seemann

I think the idea of using a random uint64 is a reasonable one and have made the change in the PR. Please let me know if there are any strong objections against this and we can discuss this in depth.

@marten-seemann the code was correct. It took a random byte string, encoded it, decoded it, then compared the bytes.

Honestly, that's a lot simpler than comparing a 64bit number as we're not relying on encoding order at all.

However, sending a raw number also works and I agree that 32 bytes was overkill for our purposes.

marten-seemann · 2021-01-22T02:15:06Z

+		return "", false, err
+	}
+
+	peernonce, err := base64.StdEncoding.DecodeString(tok[7:])


Suggested change

peernonce, err := base64.StdEncoding.DecodeString(tok[7:])

peernonce, err := base64.StdEncoding.DecodeString(tok[len(tieBreakerPrefix):])

This is done.

marten-seemann · 2021-01-22T02:15:52Z

+	case 0:
+		return "", false, errors.New("failed client selection; identical nonces!")
+
+	default:


I don't think you need a default when using bytes.Compare.

This has been removed.

marten-seemann · 2021-01-22T02:20:12Z

+		// peer nonce bigger, he is client
+		iamserver = true
+
+	case 1:


You actually don't need this case, iamserver is already initialized to false.

I think this helps readability of the code a bit, so keeping it around.

marten-seemann · 2021-01-22T02:21:06Z

+	if err != nil {
+		return "", err
+	}
+	if tok != "initiator" {


Maybe make this a const? I see you're using it a few times.

This is done for both initiator and responder.

aarshkshah1992

@marten-seemann

Have addressed all your review comments. Thanks !

aarshkshah1992 · 2021-01-22T10:24:25Z

+		// peer nonce bigger, he is client
+		iamserver = true
+
+	case 1:


I think this helps readability of the code a bit, so keeping it around.

aarshkshah1992 · 2021-01-22T10:24:33Z

+	case 0:
+		return "", false, errors.New("failed client selection; identical nonces!")
+
+	default:


This has been removed.

aarshkshah1992 · 2021-01-22T10:24:44Z

+	if err != nil {
+		return "", err
+	}
+	if tok != "initiator" {


This is done for both initiator and responder.

aarshkshah1992 · 2021-01-22T10:25:11Z

+		return "", false, err
+	}
+
+	peernonce, err := base64.StdEncoding.DecodeString(tok[7:])


This is done.

aarshkshah1992 · 2021-01-22T10:47:10Z

+	}
+
+	var iamserver bool
+	switch bytes.Compare(mynonce, peernonce) {


@vyzo @marten-seemann

I think the idea of using a random uint64 is a reasonable one and have made the change in the PR. Please let me know if there are any strong objections against this and we can discuss this in depth.

marten-seemann · 2021-01-22T14:50:32Z

+		iamserver = true
+	} else if peerNone < myNonce {
+		// my nonce bigger, i am client
+		iamserver = false


Probably nicer:

if peerNonce == myNonce { return ... } iamserver := peerNonce > myNonce

marten-seemann · 2021-01-22T14:50:53Z

+		return "", false, err
+	}
+
+	peerNone, err := strconv.ParseUint(tok[len(tieBreakerPrefix):], 10, 64)


Typo? peerNonce?

aarshkshah1992 · 2021-01-25T10:02:40Z

@marten-seemann @Stebalien

Have addressed all review comments so far. Please approve if happy and we can ship !

Stebalien

feel free to ignore the nits.

Stebalien · 2021-02-12T06:06:13Z

+	}
+
+	switch tok {
+	case "iamclient":


nit: define a string constant.

Stebalien · 2021-02-12T06:07:46Z

+		if err != nil {
+			return "", false, err
+		}


nit: do we need this if err? Or can we just return proto, false, err.

Stebalien · 2021-02-12T06:09:27Z

+	case "na":
+		return selectProtosOrFail(protos[1:], rwc)
+	default:
+		return "", errors.New("unexpected response: " + tok)


nit: fmt.Errorf("unexpected response: %s", tok) is technically more idiomatic (although it really doesn't matter).

Stebalien · 2021-02-12T06:14:24Z

+	randBytes := make([]byte, 8)
+	_, err := rand.Read(randBytes)
+	if err != nil {
+		return "", false, err
+	}
+	myNonce := binary.LittleEndian.Uint64(randBytes)


nit: it's probably slightly simpler to do:

var myNonce uint64 if err := binary.Read(rand.Reader, binary.LittleEndian, &myNonce); err != nil { ... } ``

Stebalien · 2021-02-12T06:16:52Z

+		return "", false, err
+	}
+
+	var iamserver bool


nit: drop this line.

Stebalien · 2021-02-12T06:20:46Z

+		tok, err = ReadNextToken(rwc)
+
+		if err == io.EOF {
+			return "", ErrNotSupported


EOF will happen when the remote side gives up because we have no protocols in common. In general, it's best to avoid returning EOF for actual error cases anyways (usually, you want to either return ErrUnexpectedEOF, or a better error like we're doing here).

vyzo · 2021-03-11T17:45:07Z

For the record, I find the use of a uint strictly worse than the original spec from a protocol standpoint; but then again it doesn't matter much at the end of the day.
We do need to update the spec for what was actually implemented however.

The Golang implementation does not retry. See multiformats/go-multistream#42 (comment).

vyzo requested review from Stebalien and raulk August 26, 2019 11:15

vyzo mentioned this pull request Aug 26, 2019

NAT traversal: Implement support for simultaneous open in mulstistream libp2p/go-libp2p#712

Closed

5 tasks

vyzo force-pushed the feat/simopen branch from 07f61fb to 2bc1827 Compare August 26, 2019 12:42

vyzo force-pushed the feat/simopen branch from 2bc1827 to 36a530c Compare September 21, 2019 09:21

raulk reviewed Sep 24, 2019

View reviewed changes

aarshkshah1992 reviewed Jan 7, 2021

View reviewed changes

vyzo added 5 commits January 12, 2021 10:44

implement simultaneous open extension

0a5c7a1

handle protcol mismatch correctly in server path of simultaneous open

5da27f8

add simultaneous open tests

276e57e

concurrent read/write goroutines for simultaneous open

ef959da

encode simopen selection nonces in base64

b087b1b

aarshkshah1992 force-pushed the feat/simopen branch 2 times, most recently from d06dbd1 to b087b1b Compare January 12, 2021 08:23

Stebalien reviewed Jan 16, 2021

View reviewed changes

Comment thread client.go

aarshkshah1992 reviewed Jan 16, 2021

View reviewed changes

aarshkshah1992 added 2 commits January 16, 2021 13:05

changes as per review

2c5076b

skip exactly once protocol

85f04aa

aarshkshah1992 mentioned this pull request Jan 21, 2021

Project Flare(decentralised Hole Punching) Phase1 Meta Issue libp2p/go-libp2p#1039

Closed

23 tasks

fixed formatting

31c07c1

marten-seemann reviewed Jan 22, 2021

View reviewed changes

changes as per review

890f066

aarshkshah1992 reviewed Jan 22, 2021

View reviewed changes

marten-seemann reviewed Jan 22, 2021

View reviewed changes

fix code

4c3567f

aarshkshah1992 mentioned this pull request Jan 27, 2021

Finish and Test the simultaneous connect problem in libp2p peers libp2p/go-libp2p#1041

Merged

1 task

Stebalien approved these changes Feb 12, 2021

View reviewed changes

aarshkshah1992 merged commit 4661b85 into master Feb 12, 2021

aarshkshah1992 deleted the feat/simopen branch February 12, 2021 06:42

mxinden mentioned this pull request May 6, 2021

multistream-select 1.0: simultaneous open protocol extension libp2p/specs#196

Merged

mxinden added a commit to libp2p/specs that referenced this pull request May 7, 2021

connections/simopen: Do not retry initiator selection

ae86e33

The Golang implementation does not retry. See multiformats/go-multistream#42 (comment).

Stebalien mentioned this pull request May 11, 2021

Release v0.14.0 libp2p/go-libp2p#1090

Closed

27 tasks

aschmahmann mentioned this pull request May 14, 2021

Release v0.9.0 ipfs/kubo#8058

Closed

71 tasks

	peernonce, err := base64.StdEncoding.DecodeString(tok[7:])
	peernonce, err := base64.StdEncoding.DecodeString(tok[len(tieBreakerPrefix):])

Conversation

vyzo commented Aug 26, 2019 • edited by aarshkshah1992 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vyzo commented Sep 21, 2019

Uh oh!

raulk left a comment

Choose a reason for hiding this comment

Uh oh!

vyzo commented Sep 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aarshkshah1992 commented Jan 12, 2021

Uh oh!

Stebalien left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aarshkshah1992 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aarshkshah1992 commented Jan 21, 2021

Uh oh!

marten-seemann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vyzo commented Aug 26, 2019 •

edited by aarshkshah1992

Loading

aarshkshah1992 left a comment •

edited

Loading