You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Recent work in the SDK aimed at breaking apart the monolithic root go module has highlighted
14
-
shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common
14
+
shortcomings and inconsistencies in our testing paradigm. This ADR clarifies a common
15
15
language for talking about test scopes and proposes an ideal state of tests at each scope.
16
16
17
17
## Context
18
18
19
19
[ADR-053: Go Module Refactoring](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-053-go-module-refactoring.md) expresses our desire for an SDK composed of many
20
20
independently versioned Go modules, and [ADR-057: App Wiring Part I](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-057-app-wiring-1.md) offers a methodology
21
-
for breaking apart inter-module dependencies through the use of dependency injection. As
21
+
for breaking apart inter-module dependencies through the use of dependency injection. As
22
22
described in [EPIC: Separate all SDK modules into standalone go modules](https://github.com/cosmos/cosmos-sdk/issues/11899), module
23
23
dependencies are particularly complected in the test phase, where simapp is used as
24
-
the key test fixture in setting up and running tests. It is clear that the successful
24
+
the key test fixture in setting up and running tests. It is clear that the successful
25
25
completion of Phases 3 and 4 in that EPIC require the resolution of this dependency problem.
26
26
27
27
In [EPIC: Unit Testing of Modules via Mocks](https://github.com/cosmos/cosmos-sdk/issues/12398) it was thought this Gordian knot could be
28
28
unwound by mocking all dependencies in the test phase for each module, but seeing how these
29
29
refactors were complete rewrites of test suites discussions began around the fate of the
30
-
existing integration tests. One perspective is that they ought to be thrown out, another is
30
+
existing integration tests. One perspective is that they ought to be thrown out, another is
31
31
that integration tests have some utility of their own and a place in the SDK's testing story.
32
32
33
33
Another point of confusion has been the current state of CLI test suites, [x/auth](https://github.com/cosmos/cosmos-sdk/blob/0f7e56c6f9102cda0ca9aba5b6f091dbca976b5a/x/auth/client/testutil/suite.go#L44-L49) for
34
-
example. In code these are called integration tests, but in reality function as end to end
35
-
tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify
34
+
example. In code these are called integration tests, but in reality function as end to end
35
+
tests by starting up a tendermint node and full application. [EPIC: Rewrite and simplify
36
36
CLI tests](https://github.com/cosmos/cosmos-sdk/issues/12696) identifies the ideal state of CLI tests using mocks, but does not address the
37
37
place end to end tests may have in the SDK.
38
38
@@ -43,8 +43,8 @@ ideal state in the SDK.
43
43
### Unit tests
44
44
45
45
Unit tests exercise the code contained in a single module (e.g. `/x/bank`) or package
46
-
(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two
47
-
levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on
46
+
(e.g. `/client`) in isolation from the rest of the code base. Within this we identify two
47
+
levels of unit tests, *illustrative* and *journey*. The definitions below lean heavily on
*Illustrative* tests exercise an atomic part of a module in isolation - in this case we
@@ -65,11 +65,11 @@ supplied to the keeper constructor.
65
65
66
66
#### Limitations
67
67
68
-
Certain modules are tightly coupled beyond the test phase. A recent dependency report for
68
+
Certain modules are tightly coupled beyond the test phase. A recent dependency report for
69
69
`bank -> auth` found 274 total usages of `auth` in `bank`, 50 of which are in
70
-
production code and 224 in test. This tight coupling may suggest that either the modules
70
+
production code and 224 in test. This tight coupling may suggest that either the modules
71
71
should be merged, or refactoring is required to abstract references to the core types tying
72
-
the modules together. It could also indicate that these modules should be tested together
72
+
the modules together. It could also indicate that these modules should be tested together
73
73
in integration tests beyond mocked unit tests.
74
74
75
75
In some cases setting up a test case for a module with many mocked dependencies can be quite
@@ -82,18 +82,18 @@ Integration tests define and exercise relationships between an arbitrary number
82
82
and/or application subsystems.
83
83
84
84
Wiring for integration tests is provided by `depinject` and some [helper code](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/testutil/sims/app_helpers.go#L95) starts up
85
-
a running application. A section of the running application may then be tested. Certain
85
+
a running application. A section of the running application may then be tested. Certain
86
86
inputs during different phases of the application life cycle are expected to produce
87
-
invariant outputs without too much concern for component internals. This type of black box
87
+
invariant outputs without too much concern for component internals. This type of black box
88
88
testing has a larger scope than unit testing.
89
89
90
90
Example 1 [client/grpc_query_test/TestGRPCQuery](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/client/grpc_query_test.go#L111-L129) - This test is misplaced in `/client`,
91
91
but tests the life cycle of (at least) `runtime` and `bank` as they progress through
92
-
startup, genesis and query time. It also exercises the fitness of the client and query
92
+
startup, genesis and query time. It also exercises the fitness of the client and query
93
93
server without putting bytes on the wire through the use of [QueryServiceTestHelper](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/baseapp/grpcrouter_helpers.go#L31).
94
94
95
95
Example 2 `x/evidence` Keeper integration tests - Starts up an application composed of [8
96
-
modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite
96
+
modules](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/testutil/app.yaml#L1) with [5 keepers](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/keeper_test.go#L101-L106) used in the integration test suite. One test in the suite
97
97
exercises [HandleEquivocationEvidence](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/x/evidence/keeper/infraction_test.go#L42) which contains many interactions with the staking
98
98
keeper.
99
99
@@ -103,49 +103,49 @@ YAML as above) [statically](https://github.com/cosmos/cosmos-sdk/blob/main/x/nft
103
103
#### Limitations
104
104
105
105
Setting up a particular input state may be more challenging since the application is
106
-
starting from a zero state. Some of this may be addressed by good test fixture
107
-
abstractions with testing of their own. Tests may also be more brittle, and larger
106
+
starting from a zero state. Some of this may be addressed by good test fixture
107
+
abstractions with testing of their own. Tests may also be more brittle, and larger
108
108
refactors could impact application initialization in unexpected ways with harder to
109
-
understand errors. This could also be seen as a benefit, and indeed the SDK's current
109
+
understand errors. This could also be seen as a benefit, and indeed the SDK's current
110
110
integration tests were helpful in tracking down logic errors during earlier stages
111
111
of app-wiring refactors.
112
112
113
113
### Simulations
114
114
115
115
Simulations (also called generative testing) are a special case of integration tests where
116
116
deterministically random module operations are executed against a running simapp, building
117
-
blocks on the chain until a specified height is reached. No *specific* assertions are
117
+
blocks on the chain until a specified height is reached. No *specific* assertions are
118
118
made for the state transitions resulting from module operations but any error will halt and
119
-
fail the simulation. Since `crisis` is included in simapp and the simulation runs
119
+
fail the simulation. Since `crisis` is included in simapp and the simulation runs
120
120
EndBlockers at the end of each block any module invariant violations will also fail
121
121
the simulation.
122
122
123
123
Modules must implement [AppModuleSimulation.WeightedOperations](https://github.com/cosmos/cosmos-sdk/blob/2bec9d2021918650d3938c3ab242f84289daef80/types/module/simulation.go#L31) to define their
124
-
simulation operations. Note that not all modules implement this which may indicate a
124
+
simulation operations. Note that not all modules implement this which may indicate a
125
125
gap in current simulation test coverage.
126
126
127
-
Modules not returning simulation operations:
127
+
Modules not returning simulation operations:
128
128
129
-
-`auth`
130
-
-`capability`
131
-
-`evidence`
132
-
-`mint`
133
-
-`params`
129
+
*`auth`
130
+
*`capability`
131
+
*`evidence`
132
+
*`mint`
133
+
*`params`
134
134
135
135
A separate binary, [runsim](https://github.com/cosmos/tools/tree/master/cmd/runsim), is responsible for kicking off some of these tests and
136
136
managing their life cycle.
137
137
138
138
#### Limitations
139
139
140
-
-[A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI.
141
-
-[Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why.
142
-
- Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run
140
+
*[A success](https://github.com/cosmos/cosmos-sdk/runs/7606931983?check_suite_focus=true) may take a long time to run, 7-10 minutes per simulation in CI.
141
+
*[Timeouts](https://github.com/cosmos/cosmos-sdk/runs/7606932295?check_suite_focus=true) sometimes occur on apparent successes without any indication why.
142
+
* Useful error messages not provided on [failure](https://github.com/cosmos/cosmos-sdk/runs/7606932548?check_suite_focus=true) from CI, requiring a developer to run
143
143
the simulation locally to reproduce.
144
144
145
145
### E2E tests
146
146
147
147
End to end tests exercise the entire system as we understand it in as close an approximation
148
-
to a production environment as is practical. Presently these tests are located at
148
+
to a production environment as is practical. Presently these tests are located at
149
149
[tests/e2e](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e) and rely on [testutil/network](https://github.com/cosmos/cosmos-sdk/tree/main/testutil/network) to start up an in-process Tendermint node.
150
150
151
151
#### Limitations
@@ -164,13 +164,13 @@ The scope of e2e tests has been complected with command line interface testing.
164
164
We accept these test scopes and identify the following decisions points for each.
165
165
166
166
| Scope | App Fixture | Mocks? |
167
-
|-------------|-------------|--------|
167
+
|-----------|-----------|------|
168
168
| Unit | None | Yes |
169
169
| Integration | depinject | Some |
170
-
| Simulation |simapp | No |
170
+
| Simulation |depinject| No |
171
171
| E2E | simapp | No |
172
172
173
-
####Unit Tests
173
+
### Unit Tests
174
174
175
175
All modules must have mocked unit test coverage.
176
176
@@ -185,11 +185,11 @@ production code.
185
185
186
186
When module unit test introduction as per [EPIC: Unit testing of modules via mocks](https://github.com/cosmos/cosmos-sdk/issues/12398)
187
187
results in a near complete rewrite of an integration test suite the test suite should be
188
-
retained and moved to `/tests/integration`. We accept the resulting test logic
188
+
retained and moved to `/tests/integration`. We accept the resulting test logic
189
189
duplication but recommend improving the unit test suite through the addition of
190
190
illustrative tests.
191
191
192
-
####Integration Tests
192
+
### Integration Tests
193
193
194
194
All integration tests shall be located in `/tests/integration`, even those which do not
195
195
introduce extra module dependencies.
@@ -199,11 +199,11 @@ modules in application startup, i.e. don't depend on simapp.
199
199
200
200
Integration tests should outnumber e2e tests.
201
201
202
-
####Simulations
202
+
### Simulations
203
203
204
-
Simulations shall startup and test simapp directly.
204
+
Simulations shall use `depinject`. They are located under `/x/{moduleName}/simulation`.
205
205
206
-
####E2E Tests
206
+
### E2E Tests
207
207
208
208
Existing e2e tests shall be migrated to integration tests by removing the dependency on the
209
209
test network and in-process Tendermint node to ensure we do not lose test coverage.
@@ -220,31 +220,31 @@ demonstrated in [PR#12706](https://github.com/cosmos/cosmos-sdk/pull/12706).
220
220
221
221
### Positive
222
222
223
-
- test coverage is increased
224
-
- test organization is improved
225
-
- reduced dependency graph size in modules
226
-
- simapp removed as a dependency from modules
227
-
- inter-module dependencies introduced in test code are removed
228
-
- reduced CI run time after transitioning away from in process Tendermint
223
+
* test coverage is increased
224
+
* test organization is improved
225
+
* reduced dependency graph size in modules
226
+
* simapp removed as a dependency from modules
227
+
* inter-module dependencies introduced in test code are removed
228
+
* reduced CI run time after transitioning away from in process Tendermint
229
229
230
230
### Negative
231
231
232
-
- some test logic duplication between unit and integration tests during transition
233
-
- test written using dockertest DX may be a bit worse
232
+
* some test logic duplication between unit and integration tests during transition
233
+
* test written using dockertest DX may be a bit worse
234
234
235
235
### Neutral
236
236
237
-
- learning curve for BDD style tests
238
-
- some discovery required for e2e transition to dockertest
237
+
* learning curve for BDD style tests
238
+
* some discovery required for e2e transition to dockertest
239
239
240
240
## Further Discussions
241
241
242
242
It may be useful if test suites could be run in integration mode (with mocked tendermint) or
243
243
with e2e fixtures (with real tendermint and many nodes). Integration fixtures could be used
244
244
for quicker runs, e2e fixures could be used for more battle hardening.
245
245
246
-
A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847)
247
-
is in progress for unit tests demonstrating BDD.
246
+
A PoC `x/gov` was completed in PR [#12847](https://github.com/cosmos/cosmos-sdk/pull/12847)
247
+
is in progress for unit tests demonstrating BDD[Rejected].
248
248
Observing that a strength of BDD specifications is their readability, and a con is the
249
249
cognitive load while writing and maintaining, current consensus is to reserve BDD use
250
250
for places in the SDK where complex rules and module interactions are demonstrated.
0 commit comments