[C++][FlightRPC][Packaging] Update support for disabling TLS server verification for recent gRPC versions

[asfimport]

It is regarding issue from github: #9525

Output of conda list:

Name Version Build Channel
 abseil-cpp 20200923.3 h046ec9c_0 conda-forge
 aif360 0.3.0 pypi_0 pypi
 appdirs 1.4.4 pypi_0 pypi
 appnope 0.1.2 pypi_0 pypi
 arrow-cpp 3.0.0 py36h25f3d33_3_cpu conda-forge
 astunparse 1.6.3 pypi_0 pypi
 attrs 20.3.0 pypi_0 pypi
 aws-c-cal 0.4.5 hf7813a8_6 conda-forge
 aws-c-common 0.4.67 hbcf498f_0 conda-forge
 aws-c-event-stream 0.2.6 h8218164_4 conda-forge
 aws-c-io 0.8.3 h339dee7_1 conda-forge
 aws-checksums 0.1.11 h339dee7_1 conda-forge
 aws-sdk-cpp 1.8.138 h5307d9a_1 conda-forge
 backcall 0.2.0 pypi_0 pypi
 bayesian-optimization 1.2.0 pypi_0 pypi
 black 19.10b0 pypi_0 pypi
 boto3 1.17.9 pypi_0 pypi
 botocore 1.20.9 pypi_0 pypi
 brotli 1.0.9 h046ec9c_4 conda-forge
 bzip2 1.0.8 hc929b4f_4 conda-forge
 c-ares 1.17.1 hc929b4f_0 conda-forge
 ca-certificates 2020.12.5 h033912b_0 conda-forge
 cached-property 1.5.2 pypi_0 pypi
 category-encoders 2.1.0 pypi_0 pypi
 certifi 2020.12.5 py36h79c6626_1 conda-forge
 chardet 3.0.4 pypi_0 pypi
 click 7.1.2 pypi_0 pypi
 cycler 0.10.0 pypi_0 pypi
 cython 0.29.21 pypi_0 pypi
 decorator 4.4.2 pypi_0 pypi
 docutils 0.15.2 pypi_0 pypi
 flask 1.1.2 pypi_0 pypi
 future 0.18.2 pypi_0 pypi
 gflags 2.2.2 hb1e8313_1004 conda-forge
 glog 0.4.0 hb7f4fc5_3 conda-forge
 greenery 3.1 pypi_0 pypi
 grpc-cpp 1.35.0 h330f241_0 conda-forge
 grpcio 1.35.0 pypi_0 pypi
 h5py 2.10.0 pypi_0 pypi
 hpsklearn 0.1.0 pypi_0 pypi
 hyperopt 0.1.2 pypi_0 pypi
 ibm-cos-sdk 2.7.0 pypi_0 pypi
 ibm-cos-sdk-core 2.7.0 pypi_0 pypi
 ibm-cos-sdk-s3transfer 2.7.0 pypi_0 pypi
 ibm-watson-machine-learning 1.0.45 pypi_0 pypi
 idna 2.9 pypi_0 pypi
 imageio 2.9.0 pypi_0 pypi
 ipython 7.16.1 pypi_0 pypi
 ipython-genutils 0.2.0 pypi_0 pypi
 itsdangerous 1.1.0 pypi_0 pypi
 jedi 0.18.0 pypi_0 pypi
 jinja2 2.11.3 pypi_0 pypi
 jmespath 0.9.5 pypi_0 pypi
 joblib 1.0.1 pyhd8ed1ab_0 conda-forge
 jsonschema 2.6.0 pypi_0 pypi
 jsonsubschema 0.0.1 pypi_0 pypi
 keras 2.3.1 pypi_0 pypi
 keras-applications 1.0.8 pypi_0 pypi
 keras-preprocessing 1.1.2 pypi_0 pypi
 kiwisolver 1.3.1 pypi_0 pypi
 krb5 1.17.1 hddcf347_0 
 lale 0.4.13 pypi_0 pypi
 liac-arff 2.5.0 pypi_0 pypi
 libblas 3.9.0 8_openblas conda-forge
 libcblas 3.9.0 8_openblas conda-forge
 libcurl 7.71.1 h9bf37e3_8 conda-forge
 libcxx 11.0.1 habf9029_0 conda-forge
 libcxxabi 4.0.1 hcfea43d_1 
 libedit 3.1.20181209 hb402a30_0 
 libev 4.33 haf1e3a3_1 conda-forge
 libevent 2.1.10 hddc9c9b_3 conda-forge
 libffi 3.2.1 h475c297_4 
 libgfortran 5.0.0 9_3_0_h6c81a4c_18 conda-forge
 libgfortran5 9.3.0 h6c81a4c_18 conda-forge
 liblapack 3.9.0 8_openblas conda-forge
 libnghttp2 1.43.0 h07e645a_0 conda-forge
 libopenblas 0.3.12 openmp_h54245bb_1 conda-forge
 libprotobuf 3.14.0 hfd3ada9_0 conda-forge
 libssh2 1.9.0 h8a08a2b_5 conda-forge
 libthrift 0.13.0 h990abc0_6 conda-forge
 libutf8proc 2.6.1 h35c211d_0 conda-forge
 lightgbm 2.2.3 pypi_0 pypi
 llvm-openmp 11.0.1 h7c73e74_0 conda-forge
 lomond 0.3.3 pypi_0 pypi
 lxml 4.5.1 pypi_0 pypi
 lz4-c 1.9.3 h046ec9c_0 conda-forge
 markupsafe 1.1.1 pypi_0 pypi
 matplotlib 3.3.4 pypi_0 pypi
 minepy 1.2.5 pypi_0 pypi
 mock 4.0.3 pypi_0 pypi
 ncurses 6.2 h0a44026_0 
 networkx 2.4 pypi_0 pypi
 nose 1.3.7 pypi_0 pypi
 numexpr 2.7.2 pypi_0 pypi
 numpy 1.18.1 pypi_0 pypi
 opencv-python 4.5.1.48 pypi_0 pypi
 openml 0.9.0 pypi_0 pypi
 openssl 1.1.1j hbcf498f_0 conda-forge
 orc 1.6.7 h675e114_0 conda-forge
 pandas 0.25.3 pypi_0 pypi
 parquet-cpp 1.5.1 2 conda-forge
 parso 0.8.1 pypi_0 pypi
 pathspec 0.8.1 pypi_0 pypi
 patsy 0.5.1 pypi_0 pypi
 pexpect 4.8.0 pypi_0 pypi
 pickleshare 0.7.5 pypi_0 pypi
 pillow 8.1.0 pypi_0 pypi
 pip 20.0.2 py36_1 
 ply 3.11 pypi_0 pypi
 prompt-toolkit 3.0.16 pypi_0 pypi
 psutil 5.8.0 pypi_0 pypi
 ptyprocess 0.7.0 pypi_0 pypi
 pyarrow 3.0.0 py36h02a7306_3_cpu conda-forge
 pygments 2.8.0 pypi_0 pypi
 pymongo 3.10.1 pypi_0 pypi
 pyomo 5.7.3 pypi_0 pypi
 pyparsing 2.4.7 pypi_0 pypi
 python 3.6.10 h359304d_0 
 python-dateutil 2.8.1 pypi_0 pypi
 python-graphviz 0.13.2 pypi_0 pypi
 python-intervals 1.10.0 pypi_0 pypi
 python_abi 3.6 1_cp36m conda-forge
 pytz 2019.3 pypi_0 pypi
 pyutilib 6.0.0 pypi_0 pypi
 pywavelets 1.1.1 pypi_0 pypi
 pyyaml 5.4.1 pypi_0 pypi
 rbfopt 4.2.2 pypi_0 pypi
 re2 2020.11.01 h2e338ed_0 conda-forge
 readline 7.0 h1de35cc_5 
 regex 2020.11.13 pypi_0 pypi
 requests 2.23.0 pypi_0 pypi
 s3transfer 0.3.4 pypi_0 pypi
 scikit-image 0.17.2 pypi_0 pypi
 scikit-learn 0.20.3 pypi_0 pypi
 scipy 1.4.1 pypi_0 pypi
 seaborn 0.11.1 pypi_0 pypi
 setuptools 45.2.0 py36_0 
 six 1.14.0 pypi_0 pypi
 snappy 1.1.8 hb1e8313_3 conda-forge
 sqlite 3.31.1 ha441bb4_0 
 statsmodels 0.11.1 pypi_0 pypi
 tables 3.5.1 pypi_0 pypi
 tabulate 0.8.6 pypi_0 pypi
 threadpoolctl 2.1.0 pypi_0 pypi
 tifffile 2020.9.3 pypi_0 pypi
 tk 8.6.8 ha441bb4_0 
 toml 0.10.2 pypi_0 pypi
 tqdm 4.36.1 pypi_0 pypi
 traitlets 4.3.3 pypi_0 pypi
 typed-ast 1.4.2 pypi_0 pypi
 urllib3 1.25.9 pypi_0 pypi
 watson-machine-learning-client-v4 1.0.159 pypi_0 pypi
 wcwidth 0.2.5 pypi_0 pypi
 werkzeug 1.0.1 pypi_0 pypi
 wheel 0.34.2 py36_0 
 wrapt 1.12.1 pypi_0 pypi
 xgboost 0.90 pypi_0 pypi
 xlrd 2.0.1 pypi_0 pypi
 xmltodict 0.12.0 pypi_0 pypi
 xz 5.2.5 haf1e3a3_1 conda-forge
 zlib 1.2.11 h1de35cc_3 
 zstd 1.4.8 hf387650_1 conda-forge

 

Environment: macOS, conda env, python 3.6 / 3.7
Reporter: Amadeusz
Assignee: David Li / @lidavidm

Externally tracked issue: #9525

PRs and other links:

• GitHub Pull Request #9569

_{Note: This issue was originally created as ARROW-11695. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
This seems like a packaging problem. The conda environment has gRPC 1.35.0, yet Flight complains about a gRPC version less than 1.27.

@lidavidm @xhochy

[asfimport]

David Li / @lidavidm:
Hmm, at least, the nightly Conda build found support: https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=1309&view=logs&j=cf796865-97b7-5cd1-be8e-6e00ce4fd8cf&t=88ee2fb8-46fd-5c68-fde3-1c8d31ba2a5f

But the conda-forge build did not: https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/278925/logs/198 

Not sure what's up with that. Doesn't seem there are build artifacts to look at (the detection works by trying to compile a C++ snippet and seeing if it succeeds/fails, CMake saves the log somewhere IIRC).

[asfimport]

Uwe Korn / @xhochy:
The build fails with the latest gRPC 1.35 release:

check_tls_opts_132.cc: /Users/uwe/mambaforge/conda-bld/arrow-cpp-ext_1614003135390/work/cpp/src/arrow/flight/try_compile/check_tls_opts_132.cc:29:61: error: no member named 'server_verification_option' in 'grpc::experimental::TlsCredentialsOptions'
check_tls_opts_132.cc:   grpc_tls_server_verification_option server_opt = options->server_verification_option()
check_tls_opts_132.cc:                                                    ~~~~~~~  ^
check_tls_opts_132.cc: /Users/uwe/mambaforge/conda-bld/arrow-cpp-ext_1614003135390/work/cpp/src/arrow/flight/try_compile/check_tls_opts_132.cc:34:39: warning: unused variable 'opt' [-Wunused-variable]
check_tls_opts_132.cc:   grpc_tls_server_verification_option opt = check(nullptr)
check_tls_opts_132.cc:                                       ^
check_tls_opts_132.cc: 1 warning and 1 error generated.
check_tls_opts_132.cc: ninja: build stopped: subcommand failed.
check_tls_opts_132.cc:

[asfimport]

David Li / @lidavidm:
Looks like gRPC 1.35 removed the getter, so we'll have to change up how we detect this option. There's now only a setter: https://grpc.github.io/grpc/cpp/classgrpc_1_1experimental_1_1_tls_channel_credentials_options.html 

[asfimport]

David Li / @lidavidm:
Hmm. The API was completely changed in 1.34. And then the API was changed again, incompatibly, in 1.36. It's going to be hard keeping up with these changes in gRPC unless they actually stabilize, or we get community support to test & implement all the variations for different gRPC versions. We need separate implementations for 1.27, 1.32, 1.34, and (what will be) 1.36.

[asfimport]

David Li / @lidavidm:
Here's a branch with the beginnings of a fix: https://github.com/lidavidm/arrow/tree/arrow-11695-detection

But this targets the API from (unreleased) gRPC 1.36; 1.34/1.35 need more work (TlsChannelCredentialsOptions has no default constructor). It doesn't help that there is no documentation and no usage examples outside of the gRPC sources themselves. I probably won't have this finished anytime soon.

[asfimport]

Uwe Korn / @xhochy:
@lidavidm It would be nice to have a CMake option that requires this check to pass. I would set that then in the conda packages so that we are sure that it is always compiled in. Then we would at least be notified if future versions break again.

[asfimport]

David Li / @lidavidm:
Good idea. I've added an option in my branch. I'm still trying to figure out why gRPC segfaults when the new API is used, before I put up a formal PR.

[asfimport]

Uwe Korn / @xhochy:
Issue resolved by pull request 9569
#9569

[asfimport]

Amadeusz:
When we could expect a new version to be released either on pypi or conda?

[asfimport]

Uwe Korn / @xhochy:
The conda package is already fixed, on PyPI you need to wait for Arrow 4.0

[asfimport]

Amadeusz:
After new conda package installation there is a problem with certs:

E0226 09:16:14.615553000 4724039104 ssl_utils.cc:568]                  load_file: {"created":"@1614327374.615536000","description":"Failed to load file","file":"../src/core/lib/iomgr/load_file.cc","file_line":72,"filename":"/usr/share/grpc/roots.pem","referenced_errors":[{"created":"@1614327374.615533000","description":"No such file or directory","errno":2,"file":"../src/core/lib/iomgr/load_file.cc","file_line":45,"os_error":"No such file or directory","syscall":"fopen"}]}
E0226 09:16:14.615568000 4724039104 ssl_utils.cc:404]                  Could not get default pem root certs.
E0226 09:16:14.615572000 4724039104 tls_security_connector.cc:351]     Update handshaker factory failed.
E0226 09:16:14.809187000 4724039104 tls_security_connector.cc:196]     Client BlockOnInitialCredentialHandshaker not supported yet.
Traceback (most recent call last):
  File "....py", line 32, in <module>
    readClient.authenticate(token_client)
  File "pyarrow/_flight.pyx", line 1155, in pyarrow._flight.FlightClient.authenticate
  File "pyarrow/_flight.pyx", line 70, in pyarrow._flight.check_flight_status
pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: failed to connect to all addresses

[asfimport]

Uwe Korn / @xhochy:
[~amasend] please update and try again, I have pushed a new build with the final patch.

[asfimport]

Amadeusz:
@xhochy Thanks! It is working now! Great work.