[BUG] [Java] `CagraBuildAndSearchIT.testFloatIndexing` fails randomly

[mythrocks]

One of the tests introduced by #1366 seems to be failing randomly in CI.

For instance, from the CI run for #1459:

Error:  Tests run: 17, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 7.993 s <<< FAILURE! -- in com.nvidia.cuvs.CagraBuildAndSearchIT
Error:  com.nvidia.cuvs.CagraBuildAndSearchIT.testFloatIndexing -- Time elapsed: 0.186 s <<< ERROR!
java.util.concurrent.ExecutionException: 
java.lang.AssertionError: Exception while executing runnable: java.lang.RuntimeException: cuvsCagraBuild returned 0[RAFT failure at file=/tmp/conda-bld-output/bld/rattler-build_libcuvs/work/cpp/src/neighbors/detail/cagra/graph_core.cuh line=1406: Could not generate an intermediate CAGRA graph because the initial kNN graph contains too many invalid or duplicated neighbor nodes. This error can occur, for example, if too many overflows occur during the norm computation between the dataset vectors.
Obtained 6 stack frames
#1 in /opt/conda/envs/java/lib/jvm/lib/server/../../../libcuvs.so(+0x56826d) [0x7a570b4db26d]
#2 in /opt/conda/envs/java/lib/jvm/lib/server/../../../libcuvs.so: void cuvs::neighbors::cagra::detail::graph::optimize<unsigned int, raft::host_device_accessor<std::experimental::default_accessor<unsigned int>, (raft::memory_type)0> >(raft::resources const&, std::experimental::mdspan<unsigned int, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<unsigned int>, (raft::memory_type)0> >, std::experimental::mdspan<unsigned int, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<unsigned int>, (raft::memory_type)0> >, bool, bool) +0x23f9 [0x7a570bf36c19]
#3 in /opt/conda/envs/java/lib/jvm/lib/server/../../../libcuvs.so: cuvs::neighbors::cagra::index<float, unsigned int> cuvs::neighbors::cagra::detail::build<float, unsigned int, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >(raft::resources const&, cuvs::neighbors::cagra::index_params const&, std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >) +0x5f9 [0x7a570bf44009]
#4 in /opt/conda/envs/java/lib/jvm/lib/server/../../../libcuvs.so: cuvs::neighbors::cagra::build(raft::resources const&, cuvs::neighbors::cagra::index_params const&, std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >) +0x21 [0x7a570bf1d3b1]
#5 in /opt/conda/envs/java/lib/jvm/lib/server/../../../libcuvs_c.so: cuvsCagraBuild +0x50a [0x7a574799dffa]
#6 in [0x7a579c0861c7]
]
	at __randomizedtesting.SeedInfo.seed([66039A8CAFB9D3C9:449B6310296799E0]:0)
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CagraBuildAndSearchIT.runConcurrently(CagraBuildAndSearchIT.java:69)
	at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CagraBuildAndSearchIT.testIndexing(CagraBuildAndSearchIT.java:233)
	at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CagraBuildAndSearchIT.testFloatIndexing(CagraBuildAndSearchIT.java:215)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:426)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)

In the interest of unblocking CI on cuvs/main, I think it might be quickest to disable this test temporarily, while a fix is sought.

[mythrocks]

cc @ldematte.

[mythrocks]

Interesting: I'm getting a core dump for this seed (66039A8CAFB9D3C9) on my machine:

Current thread (0x00007cbcd4eb37e0):  JavaThread "pool-2-thread-3"        [_thread_in_native, id=2340369, stack(0x00007cbd2c20e000,0x00007cbd2c30e000) (1024K)]

Stack: [0x00007cbd2c20e000,0x00007cbd2c30e000],  sp=0x00007cbd2c309978,  free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libcuda.so.1+0x220890]
C  [libcuda.so.1+0x2c7e0a]
C  [libcuda.so.1+0x41430e]
C  [libcuda.so.1+0x41c39d]
C  [libcuda.so.1+0xfadbca]
C  [libcuda.so.1+0x1ce14c]
C  [libcuda.so.1+0x1ce93a]
C  [libcuda.so.1+0x1d0770]
C  [libcuda.so.1+0x345cda]
C  [libcuda.so.1+0x32e690]  cuMemcpy2DAsync_v2_ptsz+0x20
C  [libcudart.so.12+0x49835]
C  [libcudart.so.12+0x188cc]
C  [libcudart.so.12+0x5f85a]  cudaMemcpy2DAsync_ptsz+0x28a
C  [libcuvs_c.so+0x994cc]  std::unique_ptr<cuvs::neighbors::strided_dataset<std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >::value_type, std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >::index_type>, std::default_delete<cuvs::neighbors::strided_dataset<std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >::value_type, std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> >::index_type> > > cuvs::neighbors::make_strided_dataset<std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> > >(raft::resources const&, std::experimental::mdspan<float const, std::experimental::extents<long, 18446744073709551615ul, 18446744073709551615ul>, std::experimental::layout_right, raft::host_device_accessor<std::experimental::default_accessor<float const>, (raft::memory_type)0> > const&, unsigned int)+0x31c
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::nep_invoker_blob 0x00007cbd6c2e6308
j  java.lang.invoke.LambdaForm$MH+0x0000000010219c00.invoke(Ljava/lang/Object;JJJJJ)I+16 java.base@22
j  java.lang.invoke.LambdaForm$MH+0x000000001021e400.invokeExact_MT(Ljava/lang/Object;JJJJJLjava/lang/Object;)I+27 java.base@22
j  jdk.internal.foreign.abi.DowncallStub+0x000000001021a000.invoke(Ljava/lang/foreign/SegmentAllocator;Ljava/lang/foreign/MemorySegment;JLjava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;)I+190 java.base@22
j  java.lang.invoke.LambdaForm$DMH+0x000000001021a400.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)I+20 java.base@22
j  java.lang.invoke.LambdaForm$MH+0x000000001021c000.invoke(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)I+54 java.base@22
j  java.lang.invoke.LambdaForm$MH+0x000000001021cc00.invokeExact_MT(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)I+25 java.base@22
j  com.nvidia.cuvs.internal.panama.headers_h.cuvsCagraBuild(JLjava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;Ljava/lang/foreign/MemorySegment;)I+48 com.nvidia.cuvs@25.12.0
j  com.nvidia.cuvs.internal.CagraIndexImpl.build(Lcom/nvidia/cuvs/CagraIndexParams;Lcom/nvidia/cuvs/internal/CuVSMatrixInternal;)Lcom/nvidia/cuvs/internal/CagraIndexImpl$IndexReference;+150 com.nvidia.cuvs@25.12.0
j  com.nvidia.cuvs.internal.CagraIndexImpl.<init>(Lcom/nvidia/cuvs/CagraIndexParams;Lcom/nvidia/cuvs/CuVSMatrix;Lcom/nvidia/cuvs/CuVSResources;)V+42 com.nvidia.cuvs@25.12.0

I'm able to repro the failure from the description on my workstation without the seed.

[ldematte]

@mythrocks the question still stands: what are reasonable values for a dataset?
The test fails in the native side because we get float overflow -- that's because the test just generates random numbers for the float matrix, unbounded.
Should I generate numbers e.g. in the 0-1.0 range? What's a reasonable expectation here?

[ldematte]

Also, is it expected that cuvs fails in this way? Or should it normalize instead? Or ... something else?

[mythrocks]

From the user's perspective, I would think that there should be no segfaults, regardless of the values in the dataset.

I think there might be more than one cause at play here. (Not an expert: I should defer to @benfred, @cjnolet.):

1. There could be a float overflow that's causing test failures.
2. There might be a different bug that's causing the segfault. This feels to me like a missing stream-sync.

I'm going to try eliminating the first to see if the second still occurs.

[mythrocks]

the test just generates random numbers for the float matrix, unbounded... Should I generate numbers e.g. in the 0-1.0 range? What's a reasonable expectation here?

I'm a little confused here. CuVSTestCase::createFloatMatrix() creates new floats via RandomizedTest::randomFloat(), which in turn uses java.util.Random::nextFloat(). The float should already by in the [0.0f, 1.0f) range, unless I'm mistaken.

Edit: @ldematte: I'm still seeing segfaults at my end, even after setting createFloatMatrix() to create very tiny values. There is a chance that I might be barking up the wrong tree. In case you spend time on this bug, do let me know about the float values. I'm still inclined to chase the segfault.

[ldematte]

I'm a little confused here. CuVSTestCase::createFloatMatrix() creates new floats via RandomizedTest::randomFloat(), which in turn uses java.util.Random::nextFloat(). The float should already by in the [0.0f, 1.0f) range, unless I'm mistaken.

I think you are right, but let me double check.

There might be a different bug that's causing the segfault. This feels to me like a missing stream-sync.

Mmmm... I assumed (erroneously!) that the segfault was caused by the arithmetic error. Do you think it's unrelated? It might be!