Add OpenAI GPT tokenizer class - #148
Merged
Adrian Bonar (adrianwyatt) merged 5 commits intoMar 27, 2023
Merged
Conversation
Adrian Bonar (adrianwyatt)
previously approved these changes
Mar 23, 2023
Devis Lucato (dluc)
marked this pull request as ready for review
March 24, 2023 01:26
Devis Lucato (dluc)
assigned
Adrian Bonar (adrianwyatt) and unassigned
Adrian Bonar (adrianwyatt)
Devis Lucato (dluc)
requested review from
Adrian Bonar (adrianwyatt) and
Gil LaHaye (glahaye)
March 24, 2023 01:27
Devis Lucato (dluc)
force-pushed
the
dluc97-openaitokenizer
branch
from
March 24, 2023 01:41
7d22c12 to
447dc27
Compare
Devis Lucato (dluc)
force-pushed
the
dluc97-openaitokenizer
branch
from
March 24, 2023 01:45
10cba42 to
64bb726
Compare
Devis Lucato (dluc)
added
PR: ready for review
Devis Lucato (dluc)
added
the
PR: ready to merge
Adrian Bonar (adrianwyatt)
approved these changes
Mar 27, 2023
Adrian Bonar (adrianwyatt)
enabled auto-merge (squash)
March 27, 2023 22:41
Lee Miller (lemillermicrosoft)
self-requested a review
March 27, 2023 22:52
Desmond Howard (dehoward)
pushed a commit
to lemillermicrosoft/semantic-kernel
that referenced
this pull request
Jun 1, 2023
### Motivation and Context Add GPT2/GPT3 tokenizer, to allow counting tokens when using OpenAI. ### Description Add C# port of the tokenizers recommended by OpenAI. See https://platform.openai.com/tokenizer for more info.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Add GPT2/GPT3 tokenizer, to allow counting tokens when using OpenAI.
Description
Add C# port of the tokenizers recommended by OpenAI. See https://platform.openai.com/tokenizer for more info.