Can't install kerberos support due to missing python-krbV

[vijaykiran]

Apache Airflow version: 1.10.12

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: MacBook
• OS (e.g. from /etc/os-release): macOS 10.15.6
• Kernel (e.g. uname -a):19.6.0 Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 x86_6
• Install tools: pip
• Others: Python 3.8.5

What happened:

Trying to install airflow with kerberos support and it fails because python-krbV is not available anymore on pypi.

~/airflow-test via 🐍 v3.8.5 (airflow-test-669HeWt9)
❯ pip install 'apache-airflow[kerberos]'

Collecting apache-airflow[kerberos]
  Downloading apache_airflow-1.10.12-py2.py3-none-any.whl (4.7 MB)
     |████████████████████████████████| 4.7 MB 2.7 MB/s
Collecting email-validator
  Downloading email_validator-1.1.1-py2.py3-none-any.whl (17 kB)
Collecting colorlog==4.0.2
  Downloading colorlog-4.0.2-py2.py3-none-any.whl (17 kB)
Collecting requests<3,>=2.20.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 1.3 MB/s
Collecting cached-property~=1.5
  Downloading cached_property-1.5.1-py2.py3-none-any.whl (6.0 kB)
Collecting flask-admin==1.5.4
  Downloading Flask-Admin-1.5.4.tar.gz (1.7 MB)
     |████████████████████████████████| 1.7 MB 13.9 MB/s
Collecting lazy-object-proxy~=1.3
  Downloading lazy_object_proxy-1.5.1-cp38-cp38-macosx_10_14_x86_64.whl (19 kB)
Collecting python-nvd3~=0.15.0
  Downloading python-nvd3-0.15.0.tar.gz (31 kB)
Collecting configparser<3.6.0,>=3.5.0
  Downloading configparser-3.5.3-py3-none-any.whl (21 kB)
Collecting argcomplete~=1.10
  Downloading argcomplete-1.12.0-py2.py3-none-any.whl (38 kB)
Collecting pygments<3.0,>=2.0.1
  Downloading Pygments-2.6.1-py3-none-any.whl (914 kB)
     |████████████████████████████████| 914 kB 16.6 MB/s
Collecting flask-wtf<0.15,>=0.14.2
  Downloading Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
Collecting sqlalchemy~=1.3
  Downloading SQLAlchemy-1.3.19-cp38-cp38-macosx_10_14_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 11.6 MB/s
Collecting future<0.19,>=0.16.0
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 12.4 MB/s
Collecting tzlocal<2.0.0,>=1.4
  Downloading tzlocal-1.5.1.tar.gz (16 kB)
Collecting flask-caching<1.4.0,>=1.3.3
  Downloading Flask_Caching-1.3.3-py2.py3-none-any.whl (15 kB)
Collecting sqlalchemy-jsonfield~=0.9; python_version >= "3.5"
  Downloading SQLAlchemy_JSONField-0.9.0-py2.py3-none-any.whl (10 kB)
Collecting gunicorn<21.0,>=19.5.0
  Downloading gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 11.2 MB/s
Collecting jinja2<2.12.0,>=2.10.1
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 11.8 MB/s
Collecting python-dateutil<3,>=2.3
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 11.0 MB/s
Collecting cattrs~=1.0
  Downloading cattrs-1.0.0-py2.py3-none-any.whl (14 kB)
Collecting jsonschema~=3.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 7.1 MB/s
Collecting thrift>=0.9.2
  Downloading thrift-0.13.0.tar.gz (59 kB)
     |████████████████████████████████| 59 kB 11.3 MB/s
Collecting croniter<0.4,>=0.3.17
  Downloading croniter-0.3.34-py2.py3-none-any.whl (19 kB)
Collecting markdown<3.0,>=2.5.2
  Downloading Markdown-2.6.11-py2.py3-none-any.whl (78 kB)
     |████████████████████████████████| 78 kB 9.2 MB/s
Collecting flask-appbuilder~=2.2; python_version >= "3.6"
  Downloading Flask_AppBuilder-2.3.4-py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 16.0 MB/s
Collecting werkzeug<1.0.0
  Downloading Werkzeug-0.16.1-py2.py3-none-any.whl (327 kB)
     |████████████████████████████████| 327 kB 15.8 MB/s
Collecting tabulate<0.9,>=0.7.5
  Downloading tabulate-0.8.7-py3-none-any.whl (24 kB)
Collecting psutil<6.0.0,>=4.2.0
  Downloading psutil-5.7.2.tar.gz (460 kB)
     |████████████████████████████████| 460 kB 20.5 MB/s
Collecting iso8601>=0.1.12
  Downloading iso8601-0.1.12-py2.py3-none-any.whl (12 kB)
Collecting setproctitle<2,>=1.1.8
  Downloading setproctitle-1.1.10.tar.gz (24 kB)
Collecting zope.deprecation<5.0,>=4.0
  Downloading zope.deprecation-4.4.0-py2.py3-none-any.whl (10 kB)
Collecting funcsigs<2.0.0,>=1.0.0
  Downloading funcsigs-1.0.2-py2.py3-none-any.whl (17 kB)
Collecting alembic<2.0,>=1.0
  Downloading alembic-1.4.2.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 14.8 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting graphviz>=0.12
  Downloading graphviz-0.14.1-py2.py3-none-any.whl (18 kB)
Collecting json-merge-patch==0.2
  Downloading json-merge-patch-0.2.tar.gz (3.1 kB)
Collecting pendulum==1.4.4
  Downloading pendulum-1.4.4.tar.gz (74 kB)
     |████████████████████████████████| 74 kB 7.9 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting tenacity==4.12.0
  Downloading tenacity-4.12.0-py2.py3-none-any.whl (25 kB)
Collecting flask-login<0.5,>=0.3
  Downloading Flask-Login-0.4.1.tar.gz (14 kB)
Collecting attrs~=19.3
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pandas<2.0,>=0.17.1
  Downloading pandas-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl (10.6 MB)
     |████████████████████████████████| 10.6 MB 9.6 MB/s
Collecting unicodecsv>=0.14.1
  Downloading unicodecsv-0.14.1.tar.gz (10 kB)
Collecting python-slugify<5.0,>=3.0.0
  Downloading python-slugify-4.0.1.tar.gz (11 kB)
Collecting python-daemon>=2.1.1
  Downloading python_daemon-2.2.4-py2.py3-none-any.whl (35 kB)
Collecting flask<2.0,>=1.1.0
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
     |████████████████████████████████| 94 kB 8.2 MB/s
Collecting flask-swagger<0.3,>=0.2.13
  Downloading flask-swagger-0.2.14.tar.gz (8.6 kB)
Collecting dill<0.4,>=0.2.2
  Downloading dill-0.3.2.zip (177 kB)
     |████████████████████████████████| 177 kB 20.9 MB/s
Collecting pykerberos>=1.1.13; extra == "kerberos"
  Downloading pykerberos-1.2.1.tar.gz (24 kB)
Collecting requests-kerberos>=0.10.0; extra == "kerberos"
  Downloading requests_kerberos-0.12.0-py2.py3-none-any.whl (14 kB)
Collecting snakebite[kerberos]>=2.7.8; extra == "kerberos"
  Downloading snakebite-2.11.0.tar.gz (78 kB)
     |████████████████████████████████| 78 kB 12.4 MB/s
Collecting thrift-sasl>=0.2.0; extra == "kerberos"
  Downloading thrift_sasl-0.4.2.tar.gz (3.9 kB)
Collecting idna>=2.0.0
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 15.0 MB/s
Collecting dnspython>=1.15.0
  Downloading dnspython-2.0.0-py3-none-any.whl (208 kB)
     |████████████████████████████████| 208 kB 19.1 MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 16.9 MB/s
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 7.2 MB/s
Collecting wtforms
  Downloading WTForms-2.3.3-py2.py3-none-any.whl (169 kB)
     |████████████████████████████████| 169 kB 16.5 MB/s
Collecting itsdangerous
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting pytz
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 14.2 MB/s
Requirement already satisfied: setuptools>=3.0 in /Users/vijay/.local/share/virtualenvs/airflow-test-669HeWt9/lib/python3.8/site-packages (from gunicorn<21.0,>=19.5.0->apache-airflow[kerberos]) (49.2.1)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl (16 kB)
Requirement already satisfied: six>=1.5 in /Users/vijay/.local/share/virtualenvs/airflow-test-669HeWt9/lib/python3.8/site-packages (from python-dateutil<3,>=2.3->apache-airflow[kerberos]) (1.15.0)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.16.0.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 19.7 MB/s
Collecting natsort
  Downloading natsort-7.0.1-py3-none-any.whl (33 kB)
Collecting Flask-JWT-Extended<4,>=3.18
  Downloading Flask-JWT-Extended-3.24.1.tar.gz (31 kB)
Collecting Flask-OpenID<2,>=1.2.5
  Downloading Flask-OpenID-1.2.5.tar.gz (43 kB)
     |████████████████████████████████| 43 kB 5.5 MB/s
Collecting click<8,>=6.7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 3.3 MB/s
Collecting Flask-SQLAlchemy<3,>=2.4
  Downloading Flask_SQLAlchemy-2.4.4-py2.py3-none-any.whl (17 kB)
Collecting marshmallow<3.0.0,>=2.18.0
  Downloading marshmallow-2.21.0-py2.py3-none-any.whl (50 kB)
     |████████████████████████████████| 50 kB 11.8 MB/s
Collecting sqlalchemy-utils<1,>=0.32.21
  Downloading SQLAlchemy-Utils-0.36.8.tar.gz (138 kB)
     |████████████████████████████████| 138 kB 13.2 MB/s
Collecting marshmallow-enum<2,>=1.4.1
  Downloading marshmallow_enum-1.5.1-py2.py3-none-any.whl (4.2 kB)
Collecting prison<1.0.0,>=0.1.3
  Downloading prison-0.1.3-py2.py3-none-any.whl (5.8 kB)
Collecting PyJWT>=1.7.1
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting Flask-Babel<2,>=1
  Downloading Flask_Babel-1.0.0-py3-none-any.whl (9.5 kB)
Collecting colorama<1,>=0.3.9
  Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting apispec[yaml]<2,>=1.1.1
  Downloading apispec-1.3.3-py2.py3-none-any.whl (23 kB)
Collecting marshmallow-sqlalchemy<1,>=0.16.1
  Downloading marshmallow_sqlalchemy-0.23.1-py2.py3-none-any.whl (18 kB)
Collecting Mako
  Downloading Mako-1.1.3-py2.py3-none-any.whl (75 kB)
     |████████████████████████████████| 75 kB 10.2 MB/s
Collecting python-editor>=0.3
  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Collecting pytzdata>=2018.3.0.0
  Downloading pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
     |████████████████████████████████| 489 kB 17.5 MB/s
Collecting numpy>=1.15.4
  Downloading numpy-1.19.1-cp38-cp38-macosx_10_9_x86_64.whl (15.3 MB)
     |████████████████████████████████| 15.3 MB 11.8 MB/s
Collecting text-unidecode>=1.3
  Downloading text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
     |████████████████████████████████| 78 kB 12.4 MB/s
Collecting lockfile>=0.10
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting docutils
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
     |████████████████████████████████| 548 kB 14.6 MB/s
Collecting PyYAML>=5.1
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
     |████████████████████████████████| 269 kB 5.3 MB/s
Collecting cryptography>=1.3; python_version != "3.3"
  Downloading cryptography-3.1-cp35-abi3-macosx_10_10_x86_64.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 21.0 MB/s
Requirement already satisfied: protobuf>2.4.1 in /Users/vijay/.local/share/virtualenvs/airflow-test-669HeWt9/lib/python3.8/site-packages (from snakebite[kerberos]>=2.7.8; extra == "kerberos"->apache-airflow[kerberos]) (3.13.0)
Requirement already satisfied: argparse in /Users/vijay/.local/share/virtualenvs/airflow-test-669HeWt9/lib/python3.8/site-packages (from snakebite[kerberos]>=2.7.8; extra == "kerberos"->apache-airflow[kerberos]) (1.4.0)
ERROR: Could not find a version that satisfies the requirement python-krbV (from snakebite[kerberos]>=2.7.8; extra == "kerberos"->apache-airflow[kerberos]) (from versions: none)
ERROR: No matching distribution found for python-krbV (from snakebite[kerberos]>=2.7.8; extra == "kerberos"->apache-airflow[kerberos])

What you expected to happen: Installation would be successful

How to reproduce it:
pip install 'apache-airflow[kerberos]'

Anything else we need to know:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I think the problem is only because of snakebite which used python-krbv (which was python2 only). Apparently recently the PyPI package was removed (it was maintained by RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=1024834

RedHat replaced the usage of other Python-kerberos packages with GSSAPI (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/pdf/considerations_in_adopting_rhel_8/Red_Hat_Enterprise_Linux-8-Considerations_in_adopting_RHEL_8-en-US.pdf) - see chapter "17.4.2. Selected Python Kerberos packages have been replaced". We also had a lot of other issues with Snakebite and we HAVE to replace it before 2.0 I believe:

@elukey tried hard to implement encryption support for hdfs using snakebite-py3, but it turned out to be very difficult #5659 (comment) and the idea was to replace snakebite with pyarrow.

I think we should get rid of snakebite completely and replace it with pyarrow.

CC: @mik-laj @jaketf @debek - since I know this might impact us - it will only impact us seriously if we plan to use "hdfs" operator with Kerberos support, because it relies on snakebite currently. If we do, then maybe it's an opportunity to finally get it fixed.

[potiuk]

OK. it seems that we need to remove snakebite-kerberos altogether leaving the hdfs without kerberos support.

[potiuk]

Removed in #10865

[j3pic]

For those who were taken here by Google, and are not actually using Kerberos, but just want apache-airflow[all] to install without errors, all you need to do is add snakebyte-py3 to a requirements file that also includes apache-airflow[all] and install it using the --constraint parameter to pip.

EDIT: Well, nevermind. The advice above worked for me yesterday, but it doesn't work today. The original error is now reported, irrespective of whether snakebite-py3 is installed. I guess the only real solution now is to develop an alternative to Airflow from scratch, because this is just never going to work.