[Core] The test fails on ARM: TestBigtableSession.testOpenSSL

[suztomo]

The test fails on ARM machine.

CC: @Neenu1995 @kolea2

~/suztomo/java-bigtable-hbase/bigtable-client-core-parent/bigtable-client-core$ mvn test -DtrimStackTrace=false -Dtest=com.google.cloud.bigtable.grpc.TestBigtableSession
...
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.841 s <<< FAILURE! - in com.google.cloud.bigtable.grpc.TestBigtableSession
[ERROR] testOpenSSL(com.google.cloud.bigtable.grpc.TestBigtableSession)  Time elapsed: 0.129 s  <<< ERROR!
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_aarch_64, netty_tcnative_linux_aarch_64_fedora, netty_tcnative_aarch_64, netty_tcnative]
	at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
	at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:590)
	at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:136)
	at com.google.cloud.bigtable.grpc.TestBigtableSession.testOpenSSL(TestBigtableSession.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Environment details

1. Specify the API at the beginning of the title. General, Core, and Other are also allowed as types
2. OS type and version: Linux on ARM
3. Java version: Java 8
4. Version(s): master

Steps to reproduce

Run the test on the master branch (I tested with 21d99c3).

~/suztomo/java-bigtable-hbase/bigtable-client-core-parent/bigtable-client-core$ mvn test -DtrimStackTrace=false -Dtest=com.google.cloud.bigtable.grpc.TestBigtableSession
...
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.841 s <<< FAILURE! - in com.google.cloud.bigtable.grpc.TestBigtableSession
[ERROR] testOpenSSL(com.google.cloud.bigtable.grpc.TestBigtableSession)  Time elapsed: 0.129 s  <<< ERROR!
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_aarch_64, netty_tcnative_linux_aarch_64_fedora, netty_tcnative_aarch_64, netty_tcnative]

Code example

N/A

Stack trace

The entire log: https://gist.github.com/suztomo/faae717ff6a8a6e69a97629d8fa4df16

[ERROR] testOpenSSL(com.google.cloud.bigtable.grpc.TestBigtableSession)  Time elapsed: 0.129 s  <<< ERROR!
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_aarch_64, netty_tcnative_linux_aarch_64_fedora, netty_tcnative_aarch_64, netty_tcnative]
	at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
	at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:590)
	at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:136)
	at com.google.cloud.bigtable.grpc.TestBigtableSession.testOpenSSL(TestBigtableSession.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
	Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative_linux_aarch_64
		at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:226)
		at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
		... 32 more
	Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative_linux_aarch_64.so
		at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:175)
		... 33 more
		Suppressed: java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_aarch_64 in java.library.path
			at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
			at java.lang.Runtime.loadLibrary0(Runtime.java:871)
			at java.lang.System.loadLibrary(System.java:1124)
			at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
			at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:351)
			at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
			... 33 more
			Suppressed: java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_aarch_64 in java.library.path
				at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
				at java.lang.Runtime.loadLibrary0(Runtime.java:871)
				at java.lang.System.loadLibrary(System.java:1124)
				at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
				at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
				at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
				at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
				at java.lang.reflect.Method.invoke(Method.java:498)
				at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:371)
				at java.security.AccessController.doPrivileged(Native Method)
				at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:363)
				at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:341)

External references such as API reference guides

• ?

Any additional information below

Dependency tree:

~/suztomo/java-bigtable-hbase/bigtable-client-core-parent/bigtable-client-core$ mvn dependency:tree
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] -----------< com.google.cloud.bigtable:bigtable-client-core >-----------
[INFO] Building com.google.cloud.bigtable:bigtable-client-core 1.14.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ bigtable-client-core ---
[INFO] com.google.cloud.bigtable:bigtable-client-core:jar:1.14.1-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-bigtable:jar:1.21.0:compile
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  +- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.39.0:compile
[INFO] |  +- com.google.auto.value:auto-value-annotations:jar:1.7.4:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.39.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.5.1:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  +- com.google.protobuf:protobuf-java-util:jar:3.15.2:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] |  +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] |  +- io.perfmark:perfmark-api:jar:0.23.0:runtime
[INFO] |  \- io.grpc:grpc-protobuf-lite:jar:1.36.0:compile
[INFO] +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile
[INFO] +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile
[INFO] +- com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.21.0:compile
[INFO] +- com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.21.0:compile
[INFO] +- com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.21.0:compile
[INFO] +- com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.21.0:compile
[INFO] +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.9:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- commons-logging:commons-logging:jar:1.2:compile
[INFO] +- com.google.api:gax:jar:1.62.0:compile
[INFO] +- com.google.cloud:google-cloud-core:jar:1.94.1:compile
[INFO] +- com.google.guava:guava:jar:30.1-android:compile
[INFO] +- org.threeten:threetenbp:jar:1.5.0:compile
[INFO] +- com.google.auth:google-auth-library-oauth2-http:jar:0.24.0:compile
[INFO] +- com.google.auth:google-auth-library-credentials:jar:0.24.0:compile
[INFO] +- com.google.http-client:google-http-client:jar:1.39.0:compile
[INFO] +- com.google.cloud:google-cloud-core-http:jar:1.94.1:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.31.2:compile
[INFO] |  |  +- com.google.oauth-client:google-oauth-client:jar:1.31.4:compile
[INFO] |  |  \- com.google.http-client:google-http-client-apache-v2:jar:1.39.0:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.39.0:compile
[INFO] |  \- com.google.api:gax-httpjson:jar:0.79.0:compile
[INFO] +- com.google.api:api-common:jar:1.10.1:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:3.15.2:compile
[INFO] +- io.grpc:grpc-protobuf:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-api:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-alts:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-core:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-context:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-stub:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-grpclb:jar:1.36.0:runtime
[INFO] +- io.grpc:grpc-netty-shaded:jar:1.36.0:compile
[INFO] +- io.grpc:grpc-auth:jar:1.36.0:compile
[INFO] +- com.google.api:gax-grpc:jar:1.62.0:compile
[INFO] +- io.opencensus:opencensus-api:jar:0.28.0:compile
[INFO] +- io.opencensus:opencensus-contrib-grpc-util:jar:0.28.0:compile
[INFO] +- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
[INFO] +- commons-codec:commons-codec:jar:1.15:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:test
[INFO] |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:test
[INFO] +- io.grpc:grpc-testing:jar:1.36.0:test
[INFO] +- org.mockito:mockito-core:jar:3.8.0:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.10.20:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.10.20:test
[INFO] |  \- org.objenesis:objenesis:jar:3.1:test
[INFO] +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- junit:junit:jar:4.13.2:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.638 s
[INFO] Finished at: 2021-03-16T22:00:11Z
[INFO] ------------------------------------------------------------------------

Isn't io.grpc:grpc-netty-shaded:jar:1.36.0 supposed to supply the library?

~/suztomo/java-bigtable-hbase/bigtable-client-core-parent/bigtable-client-core$ jar tf $HOME/.m2/repository/io/grpc/grpc-netty-shaded/1.36.0/grpc-netty-shaded-1.36.0.jar |grep aarch64
META-INF/native/libio_grpc_netty_shaded_netty_tcnative_linux_aarch64.so

Two files above (one has "aarch_64" and the other has "aarch64"):

META-INF/native/libio_grpc_netty_shaded_netty_tcnative_linux_aarch_64.so (the error says this is missing)
META-INF/native/libio_grpc_netty_shaded_netty_tcnative_linux_aarch64.so (present in grpc-netty-shaded-1.36.0.jar)

[kolea2]

Is this a Bigtable issue? The test is importing from the shaded netty dependency and calling one of its methods, https://github.com/googleapis/java-bigtable-hbase/blob/master/bigtable-client-core-parent/bigtable-client-core/src/test/java/com/google/cloud/bigtable/grpc/TestBigtableSession.java#L70-L71. @suztomo, does this work on an isolated project with just that shaded netty dependency? If no, it may be better to raise this issue in the grpc repo.

[suztomo]

I think it’s a big table issue caused by grpc-netty-shaded.

Yes, I’ll check whether grpc-netty-shaded works or not.

[suztomo]

The problem seems beyond gRPC. I was able to reproduce the problem with netty-handler and netty-tcnative (without gRPC).

https://github.com/suztomo/netty-shaded-openssl#netty-tcnative-2034-does-not-have-correct-file

[suztomo]

The problem has been resolved in newer version of netty-tcnative netty/netty-tcnative#552 (comment)

Now it's a matter of upgrading the dependency in grpc-java. https://github.com/grpc/grpc-java/blob/v1.36.0/build.gradle#L179

I'll create a pull request there.

[suztomo]

Created: grpc/grpc-java#7979

[suztomo]

@igorbernstein2 I see this PR below (I saw your name there) added the test for the availability of OpenSSL but I don't see the (netty's) OpenSSL class used other places in this repository. Do you think the users still be able to use this library on an ARM machine (even when io.grpc.netty.shaded.io.netty.handler.ssl.OpenSSL.isAvailable() returns false?)

https://github.com/googleapis/java-bigtable-hbase/pull/1396/files (Jun 2017)

[kolea2]

This test is no longer used/in the codebase, so I'm going to close this out as obsolete.