[SPARK-48107][PYTHON] Exclude tests from Python distribution

[nchammas]

What changes were proposed in this pull request?

Change the Python manifest so that tests are excluded from the packages that are built for distribution.

Why are the changes needed?

Tests were unintentionally included in the distributions as part of #44920. See this comment.

Does this PR introduce any user-facing change?

No, since #44920 hasn't been released to any users yet.

How was this patch tested?

I built Python packages and inspected SOURCES.txt to confirm that tests were excluded:

cd python
rm -rf pyspark.egg-info || echo "No existing egg info file, skipping deletion"
python3 packaging/classic/setup.py sdist
python3 packaging/connect/setup.py sdist
find dist -name '*.tar.gz' | xargs -I _ tar xf _ --directory=dist
cd ..
open python/dist
find python/dist -name SOURCES.txt | xargs code

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

@HyukjinKwon - Instead of this, I prefer to exclude tests as follows:

$ diff python/MANIFEST-json.in python/MANIFEST-notests.in 
19c19
< recursive-include pyspark *.pyi py.typed *.json
---
> graft pyspark
31a32
> global-exclude **/tests/**

That way, if we add new files in the future that don't happen to match the glob expressions on this line they will still be picked up.

But I believe you prefer the approach taken here, so I've started with that.

[dongjoon-hyun]

+1, LGTM.

[HyukjinKwon]

Merged to master.

[HyukjinKwon]

:-)

[dongjoon-hyun]

:)