Adding needed Tokenizer's APIs#7047

tarekgh merged 7 commits into

dotnet:maindotnet/machinelearning:mainfrom

tarekgh:AddCopilotNeededFunctionalitytarekgh/machinelearning:AddCopilotNeededFunctionalityCopy head branch name to clipboard