go-ipfs ~~0.4.16~~ => 0.4.17 by daviddias · Pull Request #28 · ipfs/interop

daviddias · 2018-07-15T07:55:19Z

I'm seeing a lot of failures with go-ipfs 0.4.16

Exchanging files

Seems that now exchanging files is prone to hang until timeout

  exchange files
    ✓ connect go <-> js (88ms)
    ✓ connect js <-> js
    cat file
      ✓ go -> js: 1.02 kB (270ms)
      ✓ js -> go: 1.02 kB (192ms)
      ✓ js -> js: 1.02 kB (410ms)
      ✓ go -> js: 63.5 kB (247ms)
      ✓ js -> go: 63.5 kB (185ms)
      ✓ js -> js: 63.5 kB (393ms)
      ✓ go -> js: 65.5 kB (248ms)
      ✓ js -> go: 65.5 kB (177ms)
      ✓ js -> js: 65.5 kB (391ms)
      ✓ go -> js: 524 kB (511ms)
      ✓ js -> go: 524 kB (331ms)
      ✓ js -> js: 524 kB (772ms)
      2) go -> js: 786 kB
      ✓ js -> go: 786 kB (3114ms)
      ✓ js -> js: 786 kB (853ms)
      ✓ go -> js: 1.05 MB (6304ms)
      ✓ js -> go: 1.05 MB (410ms)
      ✓ js -> js: 1.05 MB (855ms)
      3) go -> js: 1.05 MB
      ✓ js -> go: 1.05 MB (3850ms)
      ✓ js -> js: 1.05 MB (683ms)
      4) go -> js: 4.19 MB
      ✓ js -> go: 4.19 MB (4462ms)
      ✓ js -> js: 4.19 MB (1265ms)
      5) go -> js: 8.39 MB
      ✓ js -> go: 8.39 MB (1807ms)
      ✓ js -> js: 8.39 MB (1663ms)
    get directory
      6) go -> js: depth: 5, num: 5
      ✓ js -> go: depth: 5, num: 5 (335ms)
      ✓ js -> js: depth: 5, num: 5 (674ms)
      7) go -> js: depth: 5, num: 10
      ✓ js -> go: depth: 5, num: 10 (521ms)
      ✓ js -> js: depth: 5, num: 10 (673ms)
      ✓ go -> js: depth: 5, num: 50
      ✓ js -> go: depth: 5, num: 50 (1816ms)
      ✓ js -> js: depth: 5, num: 50 (1787ms)
      ✓ go -> js: depth: 5, num: 100
      ✓ js -> go: depth: 5, num: 100 (16902ms)
      ✓ js -> js: depth: 5, num: 100 (3386ms)

PubSub

Seems that the http-api response format for PubSub was changed as well

  11) pubsub
       ascii data
         publish from Go, subscribe on Go:
     Uncaught AssertionError: expected { Object (from, seqno, ...) } to have property 'from' of undefined, but got 'QmZMGA9d2Gw3BZUdNgDp49Twuk3uhzHyRz9inSEvKsA4YL'

daviddias · 2018-07-15T08:12:42Z

@Stebalien @whyrusleeping did you run the interop tests as part of the release or any of the RCs?

richardschneider · 2018-07-15T08:34:04Z

@diasdavid My C# HTTP API tests are all passing with go-ipfs 0.4.16. In fact a few tests are now working! It now handles

multiple subscribers to a pubsub topic
wrapping a file in a directory

daviddias · 2018-07-15T12:12:32Z

Update, seems that PubSub is good. It just so happened that the exchange files test failures were interfering with the PubSub ones

daviddias · 2018-07-15T12:20:15Z

I'm only seeing failures specifically on moving files from Go to JS, the other way around works 100% of the time.

Playing with the timeouts, I can see more of the tests go=>js passing, but the length has increased in a order of magnitude or more.

n

daviddias · 2018-07-16T09:02:01Z

test/repo.js

-describe('repo', () => {
+// The repo was again changed on go-ipfs 0.4.16
+// TODO: create spec tests for repo
+describe.skip('repo', () => {


Stebalien · 2018-07-16T17:59:50Z

I did. As usual, they failed due to repo differences.

Stebalien · 2018-07-16T18:06:58Z

Seems that the http-api response format for PubSub was changed as well

The issue there was in goD.api.id(cb). It should have returned an error but instead seemed to have returned an empty ID. Not sure if this is due to a commands lib issue in golang or the js-ipfs-api (but it looks like the latter). Thoughts?

Stebalien · 2018-07-16T18:08:50Z

I'm only seeing failures specifically on moving files from Go to JS, the other way around works 100% of the time.

That's worrying. Is this new (not in go-ipfs 0.4.15)? See: ipfs/kubo#5183.

Stebalien · 2018-07-16T18:46:30Z

So, I added a go -> go (two go daemons) test and that seems fine (really fast, actually). Definitely an interop issue.

Stebalien · 2018-07-16T23:20:55Z

Found it: ipfs/js-ipfs#1445.

Basically, we're sending a block and then waiting for an EOF before sending the next block. There are two issues here:

JS should close the stream.
We should not be waiting. I implemented it that way for better back pressure but, really, that's not the right way to implement back pressure and will probably cause real-world performance issues.

magik6k · 2018-07-18T20:59:29Z

Go to JS fix merged in ipfs/kubo#5258

daviddias · 2018-07-19T06:38:47Z

Basically, we're sending a block and then waiting for an EOF before sending the next block.

I'm confused by the logic here. Are you saying you want a EOF (aka FIN packet in networks land) for each block you send out? If so, that breaks the streams abstraction as a stream that got closed/finalized should not be reused.

Go to JS fix merged in ipfs/kubo#5258

@magik6k any ETA for a release of that fix?

Stebalien · 2018-07-19T18:39:21Z

I'm confused by the logic here. Are you saying you want a EOF (aka FIN packet in networks land) for each block you send out? If so, that breaks the streams abstraction as a stream that got closed/finalized should not be reused.

I'm saying that, when go sends a block and then closes the stream (for writing), it expects the other node to also close the stream (for writing) to completely close it. This was still was still a bug in go as there's no reason to wait for the EOF from JS before sending off the next block, but JS should eventually send that EOF.

daviddias · 2018-07-28T08:40:09Z

It's all good and shiny again!!! :D

  exchange files
    ✓ connect go <-> js (70ms)
    ✓ connect js <-> js (43ms)
    cat file
      ✓ go -> js: 1.02 kB (266ms)
      ✓ js -> go: 1.02 kB (179ms)
      ✓ js -> js: 1.02 kB (400ms)
      ✓ go -> js: 63.5 kB (258ms)
      ✓ js -> go: 63.5 kB (174ms)
      ✓ js -> js: 63.5 kB (382ms)
      ✓ go -> js: 65.5 kB (239ms)
      ✓ js -> go: 65.5 kB (160ms)
      ✓ js -> js: 65.5 kB (401ms)
      ✓ go -> js: 524 kB (515ms)
      ✓ js -> go: 524 kB (317ms)
      ✓ js -> js: 524 kB (760ms)
      ✓ go -> js: 786 kB (531ms)
      ✓ js -> go: 786 kB (354ms)
      ✓ js -> js: 786 kB (794ms)
      ✓ go -> js: 1.05 MB (566ms)
      ✓ js -> go: 1.05 MB (441ms)
      ✓ js -> js: 1.05 MB (794ms)
      ✓ go -> js: 1.05 MB (540ms)
      ✓ js -> go: 1.05 MB (415ms)
      ✓ js -> js: 1.05 MB (838ms)
      ✓ go -> js: 4.19 MB (867ms)
      ✓ js -> go: 4.19 MB (954ms)
      ✓ js -> js: 4.19 MB (1298ms)
      ✓ go -> js: 8.39 MB (20867ms)
      ✓ js -> go: 8.39 MB (1959ms)
      ✓ js -> js: 8.39 MB (2033ms)
      ✓ go -> js: 67.1 MB (6775ms)
      ✓ js -> go: 67.1 MB (26464ms)
      ✓ js -> js: 67.1 MB (11281ms)
      ✓ go -> js: 134 MB (41416ms)
      ✓ js -> go: 134 MB (28170ms)
      ✓ js -> js: 134 MB (21916ms)

Note the numbers. go-> js is 2x of js-> go and js->js.

daviddias added 2 commits July 15, 2018 09:47

test: go-ipfs 0.4.16

f958dba

test: skip repo interop tests

c53635b

ghost assigned daviddias Jul 15, 2018

ghost added the status/in-progress In progress label Jul 15, 2018

daviddias mentioned this pull request Jul 15, 2018

docs: 2018 Q3 OKRs ipfs/js-ipfs#1409

Merged

test: make exchange file tests more interesting

e4b2c57

daviddias force-pushed the go0.4.16 branch from 0b62e49 to e4b2c57 Compare July 15, 2018 12:33

test: enable directory tests on windows

63d54e4

daviddias mentioned this pull request Jul 15, 2018

Fix exchange files failing tests on Windows #9

Closed

chore: fix them linters

120439f

n

daviddias commented Jul 16, 2018

View reviewed changes

daviddias mentioned this pull request Jul 16, 2018

Enable Circuit Relay Tests #27

Merged

Stebalien mentioned this pull request Jul 16, 2018

Close streams when done with them ipfs/js-ipfs#1445

Closed

Stebalien mentioned this pull request Jul 17, 2018

Closing streams and bitswap ipfs/kubo#5247

Closed

daviddias mentioned this pull request Jul 20, 2018

Deploy latest go-ipfs master to the Gateways and bootstrappers ipfs/infra#415

Closed

This was referenced Jul 21, 2018

Error: required Datastore.Spec entry missing form config file ipfs/js-ipfs-repo#163

Closed

fix: interoperability with go repo ipfs/js-ipfs-repo#173

Merged

Peer default config must have default Datastore config ipfs/js-ipfs#1460

Closed

daviddias mentioned this pull request Jul 22, 2018

Preloader gateways for js-ipfs ipfs/infra#410

Closed

feat: fingers crossed that everything is all good now

cfe5d36

daviddias changed the title ~~go-ipfs 0.4.16~~ go-ipfs ~~0.4.16~~ => 0.4.17 Jul 28, 2018

daviddias added 7 commits July 28, 2018 10:46

feat: allll gooood!

efd1b99

fix: semver use

f43379f

feat: do not drag CI to death

9f03e6d

fix: rm .only

2c43622

fix: skip repo interop tests on pin

7d22500

test: increase timeout for CI

fedfa32

test: order

04f1fd2

daviddias merged commit 6593ccd into master Jul 28, 2018

daviddias deleted the go0.4.16 branch July 28, 2018 12:53

ghost removed the status/in-progress In progress label Jul 28, 2018

alanshaw mentioned this pull request Aug 7, 2018

[wip] ci: Run ipfs/js-ipfs-http-client & ipfs/interop tests ipfs/kubo#5053

Closed

5 tasks

Conversation

daviddias commented Jul 15, 2018

Exchanging files

PubSub

Uh oh!

daviddias commented Jul 15, 2018

Uh oh!

richardschneider commented Jul 15, 2018

Uh oh!

daviddias commented Jul 15, 2018

Uh oh!

daviddias commented Jul 15, 2018

Uh oh!

daviddias Jul 16, 2018

Choose a reason for hiding this comment

Uh oh!

Stebalien commented Jul 16, 2018

Uh oh!

Stebalien commented Jul 16, 2018

Uh oh!

Stebalien commented Jul 16, 2018

Uh oh!

Stebalien commented Jul 16, 2018

Uh oh!

Stebalien commented Jul 16, 2018

Uh oh!

magik6k commented Jul 18, 2018

Uh oh!

daviddias commented Jul 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stebalien commented Jul 19, 2018

Uh oh!

daviddias commented Jul 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

daviddias commented Jul 19, 2018 •

edited

Loading

daviddias commented Jul 28, 2018 •

edited

Loading