Running Transforms 1.0.0 on Windows

[touma-I]

Can you provide a list of transforms that currently work on Windows and what is estimated timeframe for fixing the ones that fail to run on Windows ?

[touma-I]

Pre-requisite for testing: When testing the latest release 1.0.0 on Windows, we cloned the repo and used Anaconda Navigator to launch a Notebook Server on the Local machine. We create a virtual environment for testing and periodically delete and recreate the virtual environment when we see error that don't make sense

The runtime platform is as follow:
Python 3.12.7
Windows-11-10.0.22631-SP0
3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]

All the transform use pip install of their respective module. for example:
When testing the doc_chunk transform with the python runtime, the following command is used:
pip install --user data-prep-toolkit-transforms[doc_chunck]
When testing the doc_chunk transform with the ray runtime, the following command is used:
pip install --user data-prep-tookit-transforms[ray,doc_chunk]

All tests use the Notebook provided by the transform developers in the transform folder. such as: used were taken from their respective transform such as data-prep-kit/transforms/language/doc_chunk/doc_chunk-python.ipynb.

When the developer provided a notebook for each of the runtimes (pythan and ray), both notebooks were tested.

[touma-I]

Notebooks that have been tested and shown to work on Windows:

1. doc_chunk
2. doc_quality
3. html2parqet
4. pii_redactor
5. text_encoder
6. doc_id
7. ededup
8. filter
9. hap
10. profiler
11. resize
12. Tokenization

[touma-I]

Notebook that have failed to run and for which an issue was created to be addressed in release 1.0.1:
pdf2parquet- Python Runtime: data-prep-kit/transforms/language/pdf2parquet/pdf2parquet-python.ipynb
pdf2parquet- Ray Runtime: data-prep-kit/transforms/language/pdf2parquet/pdf2parquet-ray.ipynb

[touma-I]

Notebooks that have failed and require additional investigation for identifying the proper fix (Unable to provide estimate for fix release yet):
web2parquet: /data-prep-kit/transforms/universal/web2parquet/web2parquet.ipynb

[touma-I]

Notebooks that have failed that require workaround from developer. Estimate fix coming with 1.0.1
Developers are currently required to modify the notebook and pip install fasttext-wheel
lang_id: /data-prep-kit/transforms/language/lang_id/lang_id.ipynb

lang_id depends on fasttext. Developers need to first install fasttext-wheel on Windows prior to importing this module.