Improve Megatron Tokenization: streaming, reasoning_content support, HF in-memory tokenization, etc#1221

kevalmorabia97 merged 5 commits into

mainNVIDIA/Model-Optimizer:mainfrom

kmorabia/megatron-preprocess-data-improvementsNVIDIA/Model-Optimizer:kmorabia/megatron-preprocess-data-improvementsCopy head branch name to clipboard