davix x509 voms proxy support

[mschnepf]

I observed that davix provided by DIRACOS do not support x509 voms proxies. Also PKCS#8 keys do not work and need to converted to RSA keys, see stack overflow.
Is this the expected behaviour? In Grid a lot of services are using x509 voms proxies. Therefore, it is a common method for users.

[chaen]

Can you be a bit more specific as to which errors you get ?
In LHCb, we extensively use https with davix throughout the grid, and don't see any problem.

[mschnepf]

Oh sorry. Yes of course.

$ davix-get --debug --cert /tmp/myproxy --key /tmp/myproxy https://dcachewebdav-kit.gridka.de:2880/pnfs/gridka.de/belle/disk-only/TMP/user/mschnepf/ntuple_00071_job197853382_00.root   
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 41 locks.

DAVIX(core): HTTP/SSL Session caching DISABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /work/mschnepf/gbasf2/diracos/usr/bin/../lib64/libdavix.so.0, version: 0.7.6.200.3c48eab
DAVIX(chain):  -> readToFd
DAVIX(chain): request size 0
DAVIX(http): Create HttpRequest for https://dcachewebdav-kit.gridka.de:2880/pnfs/gridka.de/belle/disk-only/TMP/user/mschnepf/ntuple_00071_job197853382_00.root
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Connection problem, retry
DAVIX(http):  ->  Error when using reycling of session/redirect : cancel and try again
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Connection problem, retry
DAVIX(http):  ->  Error when using reycling of session/redirect : cancel and try again
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Connection problem, retry
DAVIX(http):  ->  Error when using reycling of session/redirect : cancel and try again
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Connection problem: eradicate session
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(chain): read size -1
DAVIX(core): Destroy HttpRequest
DAVIX(chain):  <- readToFd
DAVIX(chain): Negative result for operation: curl error (60): SSL peer certificate or SSH remote key was not OK. After 1 retry

That repeats 3 times and ends with
(Davix::HttpRequest) Error: Result curl error (60): SSL peer certificate or SSH remote key was not OK after 3 attempts

[chaen]

Sorry but I can't reproduce. I do not have permission for the file you mention, but if I do it on our dCache storage, it works

As you can see, I have a VOMS proxy

bash-4.2$ dirac-proxy-info
[...]
path         : /tmp/xmyproxy
VOMS         : True
VOMS fqan    : ['/lhcb/Role=production']
[...]

And I can get the file

bash-4.2$ davix-get  --cert /tmp/myproxy https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
Performing Read operation on: http://[2a00:139c:5:693:0:41:81:ba]:21175/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml?dcache-http-uuid=58e7efbd-6eda-4e6c-afcd-e286f2a208cc
[========================================] 100%     769B/769B         0B/s  

Which version of DIRAC/DIRACOS do you use ?

[chrisburr]

@chaen I can reproduce it on lxplus with DIRACOS 1 but not the host installation or DIRACOS 2:

[cburr@lxplus737]~% lb-dirac prod.py2 davix-get --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://eoslhcb.cern.ch/eos/lhcb/grid/prod/lhcb/MC/2016/B02DKPI.ROOT/00141258/0000/00141258_00000001_1.b02dkpi.root | pv > /dev/null
(Davix::HttpRequest) Error: HTTP 403 : Permission refused                                                                                                                                                                                                                      ]
   0 B 0:00:01 [   0 B/s] [<=>

[cburr@lxplus737]~% lb-dirac prod.py3 davix-get --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://eoslhcb.cern.ch/eos/lhcb/grid/prod/lhcb/MC/2016/B02DKPI.ROOT/00141258/0000/00141258_00000001_1.b02dkpi.root | pv > /dev/null
 796MiB 0:00:05 [ 156MiB/s] [                       <=>                                                                                                                                                                                                                        ]
 
 ]

[cburr@lxplus737]~% davix-get --cert /tmp/x509up_u$UID --key /tmp/x509up_u$UID https://eoslhcb.cern.ch/eos/lhcb/grid/prod/lhcb/MC/2016/B02DKPI.ROOT/00141258/0000/00141258_00000001_1.b02dkpi.root | pv > /dev/null
 796MiB 0:00:03 [ 247MiB/s] [                   <=>                                                                                                                                                                                                                            ]

[chaen]

Argh shoot, I was in my debug environment ! you're right !

The davix version which is distributed is still an RC. I will update that and we will see if it fixes it.

[chaen]

Doing it here: #185

[chaen]

I confirm it's fixed

[chaen@lxplus796 chaen]$ lb-dirac prod.py2 davix-get --version
Version: 0.7.6

[chaen@lxplus796 chaen]$ lb-dirac prod.py2 davix-get  --cert /tmp/x509up_u56212 https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
Performing Read operation on: http://[2a00:139c:5:693:0:41:81:ba]:24166/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml?dcache-http-uuid=4be2f75d-999f-461f-86ab-e4592b5b8dd5
[========================================] 100%     769B/769B         0B/s   

[chaen]

actually, I was stupid...
turns out that contrary to what you (or actually, I) could believe, 0.7.6.200.3c48eab is NOT a pre-release of 0.7.6 but of the next version... so actually I need to revert these changes.

And the problem really lies within davix. I've just opened an issue cern-fts/davix#71

If you really want that to work you can unset DAVIX_USE_LIBCURL, however you risk of having segfaults

[mpatrascoiu]

...the segfaults?

[chaen]

Yup :-) If you recall, we discussed the segfaults I was getting with libneon and you said that it will disappear, so that flag "solved" my problem :-)

[mpatrascoiu]

Fair enough, let's keep the problem "solved" :)

[chaen]

works for me :D