Skip to content

Add an abstract class for Tokenizer#53

Merged
bhavya01 merged 13 commits into
AI-Hypercomputer:mainfrom
bhavya01:tokenizer
Apr 26, 2024
Merged

Add an abstract class for Tokenizer#53
bhavya01 merged 13 commits into
AI-Hypercomputer:mainfrom
bhavya01:tokenizer

Update engine.build_tokenizer method to return SentencePieceTokenizer…

3d65556
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs