Skip to content

pullrequests Search Results · repo:CodeWithKyrian/tokenizers-php language:PHP

Filter by

6 results
 (108 ms)

6 results

inCodeWithKyrian/tokenizers-php (press backspace or delete to remove)

CodeWithKyrian/tokenizers-php
Make Encoding class Countable

This PR makes the Encoding class implement the Countable interface. This allows consumers to directly assume the size of the encoding (based on the number of token IDs) using the count() function, improving ...
  • CodeWithKyrian
  • Opened 
    on Feb 6
  • #6

CodeWithKyrian/tokenizers-php
Refactor Tokenizer properties and visibility

This PR simplifies the Tokenizer class by removing the redundant modelMaxLength property. This value is already accessible via the getConfig method, so maintaining a separate property was unnecessary duplication. ...
  • CodeWithKyrian
  • Opened 
    on Feb 6
  • #5

CodeWithKyrian/tokenizers-php
Implement configuration access and reconstruction for Tokenizer

This PR implements the getConfigmethod across the main Tokenizerclass and all its components (Models, Normalizers, PreTokenizers, PostProcessors, Decoders). This feature allows for on-demand retrieval ...
  • CodeWithKyrian
  • Opened 
    on Feb 6
  • #4

CodeWithKyrian/tokenizers-php
feat: add FixedLengthPreTokenizer for fixed-length tokenization

  • CodeWithKyrian
  • Opened 
    on Feb 6
  • #3

CodeWithKyrian/tokenizers-php
Require PHP 8.2 and make core tokenization types readonly

This PR updates the platform baseline to PHP 8.2 and tightens the immutability guarantees of core tokenization types. Motivation Dropping 8.1 simplifies future language-level features and keeps the library ...
  • CodeWithKyrian
  • Opened 
    on Feb 4
  • #2

CodeWithKyrian/tokenizers-php
Use Hugging Face PHP for Hub loading and simplify examples

Summary - Replace custom HubLoader HTTP+cache logic with the codewithkyrian/huggingface Hub client. - Adopt snapshot+manifest-based caching for tokenizer.json and tokenizer_config.json to avoid repeated ...
  • CodeWithKyrian
  • Opened 
    on Feb 4
  • #1