Skip to content

General github tests adoption#127

Closed
zilm13 wants to merge 51 commits intodevelopfrom
test/spec
Closed

General github tests adoption#127
zilm13 wants to merge 51 commits intodevelopfrom
test/spec

Conversation

@zilm13
Copy link
Copy Markdown
Member

@zilm13 zilm13 commented May 6, 2019

Restored

Moving to new tests from https://github.com/ethereum/eth2.0-specs/tree/dev/test_generators
Everything done and passed except 2 operations/attestations tests.

Keep WIP until we have repository with automatically built yaml tests, plus gradle test task should be updated to match this changes. Currently it's done manually.

@zilm13
Copy link
Copy Markdown
Member Author

zilm13 commented May 6, 2019

Everythng is passed. On hold until generated tests repository is online.

# Conflicts:
#	consensus/src/main/java/org/ethereum/beacon/consensus/spec/HelperFunction.java
#	test/src/test/java/org/ethereum/beacon/test/ShuffleTests.java
@zilm13
Copy link
Copy Markdown
Member Author

zilm13 commented May 7, 2019

There are a lot of things broken after merging v0.6.1:

  1. 2 tests in BLS, test generator is based on 1.6.0 version of py-ecc which was released in March and based on keccak (now changed to sha256)
  2. 2 valid operation/attestations tests. In research
  3. some SSZ static tests. In research.

Everything else is passed.

Copy link
Copy Markdown
Contributor

@mkalinin mkalinin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job, in general! 👍
Requested only a few changes.

Comment thread start/common/src/main/resources/config/spec-constants.yml Outdated
Comment thread start/common/src/main/resources/config/spec-constants.yml Outdated
Comment thread test/src/test/java/org/ethereum/beacon/test/StateTestUtils.java Outdated
Comment thread test/src/test/java/org/ethereum/beacon/test/runner/state/StateRunner.java Outdated
zilm13 added 4 commits May 7, 2019 18:40
# Conflicts:
#	consensus/src/main/java/org/ethereum/beacon/consensus/spec/HelperFunction.java
#	test/src/test/java/org/ethereum/beacon/test/ShuffleTests.java
@zilm13
Copy link
Copy Markdown
Member Author

zilm13 commented May 7, 2019

@mkalinin fixed everything according to your review

Copy link
Copy Markdown
Contributor

@mkalinin mkalinin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a couple of new things that might worth a fix. In spite of this, I think it's ready to get merged!

} catch (Exception e) {
return VerificationResult.failedResult(e.getMessage());
} catch (SpecCommons.SpecAssertionFailed e) {
String error = e.getStackTrace().length > 0 ? e.getStackTrace()[1].toString() : "SpecAssertion";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Literally, e.getStackTrace().length > 0 check doesn't guarantee that e.getStackTrace()[1] won't turn with index out of bounds; but I believe it's OK for this particular case.

Do you mind overriding getMessage for SpecAssertionFailed which would execute this line of code and as a result return error string?

DepositVerifier depositVerifier = new DepositVerifier(spec);
assert depositVerifier.verify(deposit, state).isPassed();
spec.process_deposit((MutableBeaconState) state, deposit);
} catch (SpecCommons.SpecAssertionFailed | AssertionError ex) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use assert in the spec, do we? I am thinking of some rare cases when AssertionError has been due to odd state of some object (usual way assert is made for); in that case I'd rather prefer to see this exception than get it swallowed here.

@zilm13
Copy link
Copy Markdown
Member Author

zilm13 commented May 11, 2019

Update. Currently failed:

  • BLS/testBlsSignMessage, BLS/testBlsMessageHashCompressed, requires update of hash method in test generator according to spec, issue created: Move to SHA256 ethereum/py_ecc#68 . After resolving generator should be updated too.
  • ssz_static - requires SSZ SOS implementation, blocked by PR resolving Update SSZ implementation with offsets #124
  • operations/attestations - there were a lot changes in attestations, generator should be updated first. Job started but not yet finished. Plus it will be updated not for v0.6.1, but for current spec, so should be ignored at this moment.

All tests are blocked by resolving of this issue: ethereum/consensus-spec-tests#1
current repository does not contain spec configs.

@zilm13
Copy link
Copy Markdown
Member Author

zilm13 commented May 16, 2019

ssz_static - requires SSZ SOS implementation, blocked by PR resolving #124

resolved by #132 , tests passed

@zilm13 zilm13 changed the title [WIP] General github tests adoption General github tests adoption May 22, 2019
@zilm13 zilm13 marked this pull request as ready for review May 22, 2019 18:19
@zilm13 zilm13 closed this May 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants