Skip to main content

tokenize

Function tokenize 

Source
fn tokenize(input: &str) -> Vec<String>
Expand description

Tokenize a segment, keeping >>, >, < as distinct tokens.