IOTest flaky

[LordRaydenMK]

When running ./gradlew clean test two test for IOTest fail (I attached the results). After running multiple times, sometimes only one of those tests fail.

The tests are:
IOTest. parallel execution makes all IOs start at the same time
IOTest. parallel execution preserves order for synchronous IOs

When running test from the IOTest class ony (from IntelliJ) they run just fine.

I am using: IntelliJ IDEA 2018.2.6 (Community Edition)
Build #IC-182.5107.16, built on November 6, 2018
JRE: 1.8.0_152-release-1248-b22 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Linux 4.4.0-139-generic

Let me know if you need more information.

test results.zip

[pakoito]

It is the concurrency all starts at the same level one? That one has always been flaky, and I don't understand why -.-

[LordRaydenMK]

I updated the issue with the name of the tests. Maybe someone else will figure it out.

[LordRaydenMK]

Turned off my computer and went to take a shower and now I have a theory so here I am in front of the computer again.

I am not sure I would call this flakiness because it always fails run from gradle and always succeeds when run from IDE (single class).

Running the tests using ./gradlew test runs the test in parallel on my 4 cores. So when the tests runs the other cores are busy running other tests.

So after adding org.gradle.parallel=false (previously it was true) all the tests executed successfully (using ./gradlew test).

BUILD SUCCESSFUL in 29m 38s
342 actionable tasks: 341 executed, 1 up-to-date

I would say a test that assumes too much about the environment is not a good test. What do you think @pakoito

[pakoito]

If you check the test, it uses threads per execution context and one to start them all.

https://github.com/arrow-kt/arrow/blob/master/modules/effects/arrow-effects-instances/src/test/kotlin/arrow/effects/IOTest.kt#L251-L272

It'd be good if you could please run the tests with -i, or use a try/catch to print the complete exception so we can understand it better, given that you have consistent repro :D

[LordRaydenMK]

I executed ./gradlew clean test -i > tests.txt and here is the result.
tests.txt

Then I tied running ./gradlew clean test -i --stacktrace > tests2.txt and the result is:
tests2.txt

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':arrow-effects-instances:test'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
        at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.api.GradleException: There were failing tests. See the report at: file:///home/coka/IdeaProjects/arrow/modules/effects/arrow-effects-instances/build/reports/tests/test/index.html
        at org.gradle.api.tasks.testing.AbstractTestTask.handleTestFailures(AbstractTestTask.java:612)
        at org.gradle.api.tasks.testing.AbstractTestTask.executeTests(AbstractTestTask.java:484)
        at org.gradle.api.tasks.testing.Test.executeTests(Test.java:603)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:801)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:768)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:131)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:120)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99)
        ... 31 more

java.lang.AssertionError: expected: [1, 2, 3, 4, 5, 6] but was: [1, 2, 3, 4, 6, 5]
	at io.kotlintest.matchers.MatchersKt.equalsError(Matchers.kt:89)
	at io.kotlintest.matchers.MatchersKt.shouldEqual(Matchers.kt:63)
	at io.kotlintest.matchers.MatchersKt.shouldBe(Matchers.kt:55)
	at arrow.effects.IOTest$23.invoke(IOTest.kt:272)
	at arrow.effects.IOTest$23.invoke(IOTest.kt:24)

point out to this line: order.toList() shouldBe listOf(1L, 2, 3, 4, 5, 6)

Let me know if there is anything else I can do @pakoito

[pakoito]

Can you please increase Thread.sleep(num * 20) to Thread.sleep(num * 50) and see if that fixes it?

Try even a higher number just in case, like 100 or 1000.

[LordRaydenMK]

I tried a few values for our number, let's call it X:

Running from command line: ./gradlew clean test
x = 50 -> FAILED
x = 100 -> PASSED
x = 75 -> PASSED
x = 63 -> PASSED
x = 56 -> FAILED (twice)

Running IOTest from Intellij:
x = 100 -> PASSED
x = 10 -> PASSED
x = 1 -> FAILED
x = 5 -> PASSED
x = 4 -> FAILED (twice)

[pakoito]

It's just flaky ;_____; I have my suspicion about why, and it has to do with the way suspended functions are started. What to do, what to do.

[1Jajen1]

When I tried running gradle tasks in parallel this constantly fails. The tests work fine for me from both gradle and intellij unless I have some load running in the background (like executing other gradle tasks).
The tests are run in parallel right (with maxParallelForks = Runtime.runtime.availableProcessors() in build.gradle)? That might cause issues with those tests if either the processor running it has some load, or just isn't really keeping up with the tests. However as long as this does not happen on travis I would not change a thing. Maybe updating to kotlintest 3 and defining parallel execution there might help (Don't know if there is a way to force a specific test to be run alone, but that would definitly only be available within the tests, not gradle).

[nomisRev]

I haven't seen or hear of this in a long time so I am closing this issue.

[mkobit]

I have been seeing this on master at e5054382dd7e1ee149bb1b2235458f9529894f23 every time I try and build locally. Last time, I got both of the below tests to fail:

IOTest. parallel execution makes all IOs start at the same time
IOTest. parallel mapping is done in the expected CoroutineContext

Gradle build scan: https://scans.gradle.com/s/q3le7c7gsqfw4

[nomisRev]

@mbokit thanks for the follow up and the commit hash. I'll look into this asap.