tokenator is a simple to use library that rework a string splitting it to meaningful tokens.
This library transforms a series of characters to tokens identified by input strings.
e.g.
"The itsy bitsy spider crawled up the water spout. | Down came the rain, and washed the spider out. | Out came the sun, and dried up all the rain, | and the itsy bitsy spider went up the spout again"
Splitting strings: spider, rain
Result tokens:
> "The itsy bitsy "
> "spider"
> " crawled up the water spout. | Down came the "
> "rain"
> ", and washed the "
> "spider"
> " out. | Out came the sun, and dried up all the "
> "rain"
> ", | and the itsy bitsy "
> "spider"
> " went up the spout again"
npm install tokenatorThe signatures of the available methods are:
function tokenator(source: string, ...separators: string[]): string[]
function tokenatorStream(...separators: string[]): (separatorsEncoding: string = "utf-8") => stream.TransformCode examples could be consulted within the test folder:
These functions work over multiple separators, like in the example above. The tokenatorStream() function creates a Transform stream. NodeJS Stream usage specification is available on the NodeJS official documentation.
MIT