summaryrefslogtreecommitdiff
path: root/src/util/Tokenizer.cpp
Commit message (Expand)AuthorAgeFilesLines
* chore: Replace nonstd::string_view with std::string_viewJoel Rosdahl2022-05-231-3/+3
* fix: Replace skip_last_empty with IncludeDelimiter (#1046)Orgad Shaneh2022-04-121-4/+0
* feat: Support preserving the delimiters on tokenizerOrgad Shaneh2022-04-051-0/+12
* Add mode parameter to split_into_* and TokenizerJoel Rosdahl2021-07-051-14/+24
* Add Tokenizer classJoel Rosdahl2021-06-271-0/+43