Update transformer and tokenizer in setup.py to new versions

[amit2kesari]

Due to older version of transformer it was facing issues and was asking to install rust in the docker image. When I installed rust still it was giving issues for pyo3 due to network calls.
The newer versions are building the docker image correctly!

Error traceback:

Building wheel for tokenizers (pyproject.toml): started
#12 55.96 Building wheel for tokenizers (pyproject.toml): finished with status 'error'
#12 55.97 error: subprocess-exited-with-error
#12 55.97
#12 55.97 × Building wheel for tokenizers (pyproject.toml) did not run successfully.
#12 55.97 │ exit code: 1
#12 55.97 ╰─> [48 lines of output]
#12 55.97 /tmp/pip-build-env-1s4lt8xo/overlay/lib/python3.9/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing '0.8.0.rc1' to '0.8.0rc1'
#12 55.97 self.metadata.version = self._normalize_version(self.metadata.version)
#12 55.97 running bdist_wheel
#12 55.97 running build
#12 55.97 running build_py
#12 55.97 creating build
#12 55.97 creating build/lib.linux-x86_64-cpython-39
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers
#12 55.97 copying tokenizers/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/models
#12 55.97 copying tokenizers/models/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/models
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/decoders
#12 55.97 copying tokenizers/decoders/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/decoders
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/normalizers
#12 55.97 copying tokenizers/normalizers/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/normalizers
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/pre_tokenizers
#12 55.97 copying tokenizers/pre_tokenizers/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/pre_tokenizers
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/processors
#12 55.97 copying tokenizers/processors/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/processors
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/trainers
#12 55.97 copying tokenizers/trainers/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/trainers
#12 55.97 creating build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/init.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/base_tokenizer.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/bert_wordpiece.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/byte_level_bpe.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/char_level_bpe.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib.linux-x86_64-cpython-39/tokenizers/implementations
#12 55.97 copying tokenizers/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers
#12 55.97 copying tokenizers/models/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/models
#12 55.97 copying tokenizers/decoders/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/decoders
#12 55.97 copying tokenizers/normalizers/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/normalizers
#12 55.97 copying tokenizers/pre_tokenizers/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/pre_tokenizers
#12 55.97 copying tokenizers/processors/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/processors
#12 55.97 copying tokenizers/trainers/init.pyi -> build/lib.linux-x86_64-cpython-39/tokenizers/trainers
#12 55.97 running build_ext
#12 55.97 running build_rust
#12 55.97 error: can't find Rust compiler
#12 55.97
#12 55.97 If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

[ykim362]

Thanks for the PR, @amit2kesari.
Do you know if the models work correctly with the latest transofrmers and tokenizer?

[amit2kesari]

Hello Young Jin, Yes, we run pruning and quantization job which work correctly with the newest transformer and tokenizer versions. If you wish you can try with your specific use cases/test cases as well. Regards From: Young Jin Kim ***@***.***> Date: Wednesday, 17 January 2024 at 12:20 PM To: microsoft/fastformers ***@***.***> Cc: Amit Kesari ***@***.***>, Mention ***@***.***> Subject: Re: [microsoft/fastformers] Update transformer and tokenizer in setup.py to new versions (PR #23) Thanks for the PR, @amit2kesari<https://github.com/amit2kesari>. Do you know if the models work correctly with the latest transofrmers and tokenizer? — Reply to this email directly, view it on GitHub<#23 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BBRMRUC3VCNI4MXU7REWSUDYO5X6DAVCNFSM6AAAAABBZ32HH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJVGEZTENBXHA>. You are receiving this because you were mentioned.Message ID: ***@***.***> Amit Kesari Product Engineer | Engineering - Others/Shared | Gurugram | Sprinklr<https://www.sprinklr.com/?utm_medium=email&utm_source=signatures&utm_campaign=GLOBAL____employee-email-sig> | LinkedIn<https://www.linkedin.com/in/https://www.linkedin.com/in/amitkesari2000/> Un-siloed teams. Happier customers.™ [cid:sprinklr_signature_2(1)_ceca8167-bc9e-4801-a8a0-e17733f68ed6.gif]<https://www.sprinklr.com/?utm_medium=email&utm_source=signatures&utm_campaign=GLOBAL____employee-email-sig>