[v7r3] login with tokens (first part of the #5045)

[TaykYoku]

This is the first part of the main release #5045 and attempt to implement the code with the least changes to the main project.
Here the opportunity to log in using identity providers and access to DIRAC services with the help of tokens is realized.

UPDATE:

• Added TokenManager service

TODO:

• update to use last authlib, pyjwt packages.
• add required packages to DIRACOS2
• approve puppet commit

How to test it?

Usual tests

If you are testing without the use of tokens, for example to make sure that the usual behavior has not changed without the use of token authorization, then you do not need to do anything special. Everything must be backward compatible. Just do the tests as usual.

Try tokens

• register new account in https://wlcg.cloud.cnaf.infn.it if you don't have it.
• joing to wlcg groups

DIRAC CLI test

• Install DIRAC Client with last DIRACOS2, e.g.:

curl -LO https://github.com/DIRACGrid/DIRACOS2/releases/latest/download/DIRACOS-Linux-x86_64.sh
bash DIRACOS-Linux-x86_64.sh
source ~/diracos/diracosrc
pip install git+https://github.com/TaykYoku/DIRAC.git@integration_oauth_p01

• configure client, this will create ~/diracos/etc/dirac.cfg with required options /DIRAC/Security/Authorization/issuer=https://lbcertifdiracoauth.cern.ch/auth, /DIRAC/Setup=DIRAC-CertifOauth, DIRAC/Configuration/Servers=https://lbcertifdiracoauth.cern.ch:9135/Configuration/Server (if you already have configured dirac.cfg you can add it manually):

dirac-configure --login=https://lbcertifdiracoauth.cern.ch/auth

Note If you are not yet registered in DIRAC, the first time you try to log in through DIRAC admins will receive a message asking to add you, after which you can try again.

• check installation and authorization status, when you configure script:

dirac-info
dirac-login --info

Note when you use login you get access token from Identity provide with scopes that described for DIRAC group. If you whant to get proxy as usual use --proxy key.

• try to access to https service:

dirac-dms-filecatalog-cli

• Most services work through DISET so you can try to get proxy instead of token (proxy must be uploaded into ProxyManager as usual):

dirac-login -g dirac_user --proxy

• check proxy:

dirac-login --info
dirac-proxy-info

• try to access to https service or DISET service:

dirac-proxy-get-uploaded-info
dirac-dms-filecatalog-cli

Note to use proxy to access to https services /DIRAC/Security/UseTokens CS option and DIRAC_USE_ACCESS_TOKEN environ must be not y, yes or true.

• logout, to destroy tokens.

dirac-logout

DIRAC WEB test

• Go to https://lbcertifdiracoauth.cern.ch/DIRAC
• Try to login with IAM. Click Settings --> User --> IAM_dev.
  Note If you are not yet registered in DIRAC, the first time you try to log in through DIRAC admins will receive a message asking to add you, after which you can try again.

Note: every time you login the tokens are updated by TokenManager and you can check them in the Token Manager application.
Note: The server may not be available from time to time, for temporary reasons (eg puppet)

Prepare certification server

Set values:

• Root DIRAC directory:

install_root="/opt/dirac"

• DIRAC version in vX.Y.Z-TIMESTAMP format:

version_and_date="v7.3.0a23-$(date -u '+%s')"

• platform and DIRACOS installation path:

platform="$(uname -s)-$(uname -m)"
install_diracos_root="${install_root}/versions/${version_and_date}/${platform}"

Install DIRACOS2

• Create versions path if not exist:

mkdir -p "${install_root}/versions" && cd "${install_root}"

• install DIRACOS2:

curl -LO "https://github.com/DIRACGrid/DIRACOS2/releases/latest/download/DIRACOS-${platform}.sh"
bash "DIRACOS-${platform}.sh" -p "${install_diracos_root}" && rm "DIRACOS-${platform}.sh"

source "${install_diracos_root}/diracosrc"

• create symlink for pro(use -sfn key):

ln -sfn "${install_root}/versions/${version_and_date}" "${install_root}/pro"

Install DIRAC and WebApp

• install DIRAC and WebApp

pip install git+https://github.com/TaykYoku/DIRAC.git@integration_oauth_p01
pip install git+https://github.com/TaykYoku/WebAppDIRAC.git@integration_oauth_token

Restart services:

runsvctrl t /opt/dirac/startup/*

BEGINRELEASENOTES

*Configuration
NEW: add helpers methods to work with identity providers and APIs
FIX: decode bytes to utf-8

*Core
CHANGE: add possibility to use restful endpoints to TornadoServer framework
NEW: add possibility to use token authorisation to Tornado clients

*Framework
NEW: add Authorisation Server based on authlib package and AuthDB database to serve sessions
NEW: add restful API for Authorisation Server
NEW: add dirac-login, dirac-logout scripts
NEW: add TokenManager service with TokenDB database

*Resources
NEW: add identity providers

*tests
NEW: add AuthDB, TokenDB tests

ENDRELEASENOTES

[TaykYoku]

to pass tests I need to add new packages to DIRACOS?

authlib == 1.0.0
pyjwt
dominate

[fstagni]

This is just a very quick review, I didn't analyze in depth.

[fstagni]

to pass tests I need to add new packages to DIRACOS?

authlib == 1.0.0
pyjwt
dominate

Yes.

[chaen]

It's only a partial review, it's too long for me to review at once

[fstagni]

A non-complete review

[fstagni]

What is this for?

[TaykYoku]

to get a proxy certificate instead of a token, I will add the keys

[fstagni]

Sorry, I am confused then: isn't dirac-proxy-init for the proxy, dirac-login for the token?

[TaykYoku]

dirac-proxy-init use user certificate to generate proxy with DIRAC group extension.
dirac-login use DIRAC authorization server authorization_endpoint to AuthN through IdP(IAM/CheckIn) and AuthZ:

• by obtaining access tokens with scopes that related to DIRAC group from IdP(IAM/CheckIn).
• by obtaining proxy with DIRAC group extension from DIRAC ProxyManager

[TaykYoku]

I'll think about how to combine it all.

[chrisburr]

The WLCG Bearer Token Discovery document also mentions handling multiple tokens, for cases where multiple VOs are being used. I know very little about multi-vo DIRAC but is this something that should be included in the design of this interface?

[TaykYoku]

It is not implemented yet, but it can be implemented later based on practical use

[TaykYoku]

restart tests

[TaykYoku]

I avoided using token authentication when using python 2. I hope I understood the latest requirements correctly.
Anyway, IIUC, since the next two releases are still planned the ability to install a DIRAC server with python 2, so I can create a separate PR if need to be enable OAuth2 support in such an installation.