Discussion: should the default recipe set for python3 unconditionally include libffi to build ctypes? What about sqlite3 and other core modules?

I understand that not including libffi and ctypes brings down file size, but given it's such a basic component of so many other libs and recipes (which often use it to interface with native code), I wonder if maybe it should be made part of the built-in default recipe set and always included?

Because this error isn't super intuitive for beginners, and not something one would expect for a built-in module:

[INFO]:    Building setuptools for armeabi-v7a
[INFO]:    setuptools apparently isn't already in site-packages
[INFO]:    Installing setuptools into site-packages
[INFO]:    -> directory context /root/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools
[DEBUG]:   -> running python setup.py install -O2 --root=/root/.local/share/python-for-android/build/python-installs/unnamed_dist_1 --install-lib=.
[DEBUG]:   	Traceback (most recent call last):
[DEBUG]:   	  File "setup.py", line 11, in <module>
[DEBUG]:   	    import setuptools
[DEBUG]:   	  File "/root/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools/setuptools/__init__.py", line 14, in <module>
[DEBUG]:   	    from setuptools.dist import Distribution, Feature
[DEBUG]:   	  File "/root/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools/setuptools/dist.py", line 25, in <module>
[DEBUG]:   	    from setuptools import windows_support
[DEBUG]:   	  File "/root/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools/setuptools/windows_support.py", line 2, in <module>
[DEBUG]:   	    import ctypes
[DEBUG]:   	  File "/root/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/Lib/ctypes/__init__.py", line 7, in <module>
[DEBUG]:   	    from _ctypes import Union, Structure, Array
[DEBUG]:   	ModuleNotFoundError: No module named '_ctypes'

Edit: cffi -> libffi

Adding a point to the discussion: I also find it a bit non-intuitive that sqlite3 needs a separate recipe that I need to specify.

Wouldn't a mechanism be better where all standard modules that we can build with (including openssl) and all their according recipes are enabled per default, and can be disabled explicitly? Like, maybe some sort of recipe blacklist?

IMHO python isn't really suitable for tiny apps anyway, so I can't speak for everyone else but my app is already so large (50MB) that having or not having sqlite3 or openssl really won't make much of a difference. However, it makes a big difference for anyone coming from desktop to python-for-android and suddenly finding core modules unexpectedly missing

Another point: if we move away from --no-deps in build.py, move to running the project's setup.py per default, and picking up pip recipes that match recipes automatically, then this is also a problem because a regular application would have none of the standard stuff as dependencies (since on desktop/all other platforms, it's expected to be "just there"). So this steers us towards another weird long-term rift with IMHO no large need, when what I feel like we really should be doing is get closer to the desktop experience and a standard setup.py-powered process with standard pip dependency resolution, at least regarding p4a's default behavior

[hackalog]

Yeah, I really think libffi, openssl, and sqlite3 should be included in the build. If space is an issue, someone should make a "tinypython" recipe to remove it.

[hackalog]

I think the real bug right now is that order is critical in the requirements line, and that seems just wrong. (e.g. sqlite before python). p4a uses a dependency graph, as does pip, so order should not matter there.

I like the idea of having two different recipes, a corepython3/tinypython3 and depending on it, python3 maybe? This way there'd be a documented way to get a minimal build, but anyone just putting python3 will have all the features without bothering to manually add openssl or sqlite (which would then just be regular dependencies which wouldn't need to be specified in any way).

The alternate approach would be to have a recipe blacklist option. But I think I like tinypython3/corepython3 vs regular python3 better, personally

[RobertFlatt]

I tried to build all the user modules for python3, kivy. Of course a successful
apk build does not necessarily imply the apk contents are correct!

The following dependencies are required in addition to those for a simple .py build
(for which I use https://groups.google.com/forum/#!topic/kivy-users/Lf4zlYmLVPo )
sudo apt-get install -y autoconf libtool cmake libffi-dev pkg-config

I use python-for-android (master, installed 1/12/2019)
I tested each with a placeholder main.py and:
p4a apk --requirements=<module>,python3,kivy --arch=armeabi-v7a --ndk_version 17c --android_api 28 --ndk-api 21

The following modules successfully build an apk:

['Pillow', 'audiostream', 'cffi', 'cryptography', 'enum34', 'ffmpeg',
'ffpyplayer_codecs', 'freetype', 'harfbuzz', 'idna', 'ipaddress', 'jedi',
'jpeg', 'leveldb', 'libcurl', 'libexpat', 'libffi', 'libiconv', 'libogg',
'libsecp256k1', 'libshine', 'libvorbis', 'libx264', 'libxml2', 'libxslt',
'lxml','numpy', 'openal','openssl', 'pbkdf2', 'png', 'pycparser',
'pycryptodome', 'pyjnius', 'pysdl2','requests', 'sdl2_image', 'sdl2_mixer',
'sdl2_ttf', 'six', 'snappy', 'spine','sqlite3']

BUT, of these, the following fail with error ".. _ctypes..""
if module order is changed to --requirements=python3,kivy,<module>

['Pillow','cffi','cryptography','enum34','idna','ipaddress','pbkdf2',
'pycparser','pycryptodome','requests','spine']

Which confirms that order can be important.

The following do not build because they have no dependency on 'python3', but
do have a dependency on some combination of 'python2'/'hostpython2'/
'python3crystax'/'hostpython3'

['apsw','atom','babel','boost','cdecimal','cherrypy','coverage','cymunk',
'dateutil','decorator','doubleratchet','enaml','ethash','evdev','feedparser',
'gevent','gevent-websocket','greenlet','groestlcoin_hash','icu','ifaddrs',
'kiwisolver','libglob','libnacl','libpq','libsodium','libtorrent',
'libtribler','libzbar','libzmq','m2crypto','msgpack-python','netifaces',
'omemo','omemo-backend-signal','pil','preppy','psycopg2','pyaml','pyasnl',
'pycrypto','pyethereum','pyicu','pyleveldb','pymunk','pynacl','pyogg',
'pyopenal','pyopenssl','pyproj','pyrxp','pysha3','pyusb','pyyaml','pyzmq',
'regex','ruamel.yaml','scrypt','secp256k1','shapely','simple-crypt',
'sqlalchemy','storm','sympy','ujson','vispy','websocket-client','wsaccl',
'x3dh','xeddsa','zbar','zeroconf','zope','zope_interface']

The following does not build because of a dependency I can't identify:
['flask']

The following do not build because they have a dependency on 'kivy' !
['kivent_core','kivent_cymunk','kivent_particles','kivent_polygen']

The following does not build because:
'p4a.cmake': No such file or directory
['libmysqlclient']

The following does not build because of cmake errors
['opencv']

The following does not build because (seems strange):
crypt.h: No such file or directory
['reportlab']

The following does not build because has dependency on 'incremental', 'constantly'
['twisted']

The following does not build because NDK v18 needed
['vlc']

@Ham-Merhead was this supposed to go here by any chance? #1514

[inclement]

Do we know what is the actual size increase if including sqlite3 and openssl? Both the extracted size, and for the apk itself?

[RobertFlatt]

10702338        dir_sqlite3/sqlite3-0.0.0-debug.apk
11547402        dir_openssl/openssl-0.0.0-debug.apk
10311266        dir_REF/REF-0.0.0-debug.apk

15725333        dir_sqlite3/assets/_python_bundle
15782393        dir_openssl/assets/_python_bundle
15654564        dir_REF/assets/_python_bundle

  5317444        dir_sqlite3/lib
  7246976        dir_openssl/lib
  4784236        dir_REF/lib