The first release of the scan library (0.1.0) has been published, which parses text into values based on a template known at compile time. The template or format is written as a C++ template argument, transforming into a tagged deterministic finite automaton (TDFA) at compile time, which is traversed by states in the code using C++ templates. The result is a value of the requested type. The project code is written in C++23 and distributed under the GPLv3 license. import scan; // Does the entire string match? Each entry point is also a range adapter. scan::match(address); address | scan::match; // With groups. const auto found = scan::match("42-abc"); found.get().to_view(); // "42" // Beginning of the string occupied by the template and the first match anywhere. scan::starts_with("abc123").whole().to_view(); // "abc" scan::search("id=4210x").to_view(); // "4210" // All matches and pieces between them - lazy representations. for (const auto& one : text | scan::search_all) { … } const auto fields = "a,bb,,ccc" | scan::split | std::ranges::to(); // Values, not text. struct row { int id; std::string_view name; }; const row one = scan::scan(line); // List, sum types, nested form. struct all { std::vector values; std::variant tail; }; const all got = scan::scan("1,2,3 abc"); // Start of input and what is left from it. const auto [value, rest] = scan::scan_prefix(line).take(); // By record at a time, from anything. for (const row& one : scan::each(text).of()) { … }
A range that can be read only once is read without buffering, with fields being collected as characters arrive. The number of characters to retain is specified by the template at compile time; where such a number does not exist, reading is rejected at compile time. std::istringstream source("set speed 42\nset gain 7\n"); source > std::noskipws; struct command { scan::held name; int value; }; for (const command& one : scan::each(std::views::istream(source)) .of()) { … } Performance.
Measurements were taken on Ryzen 9 9950X, clang 22.1.8 with libc++, -O3 -march=native, LTO. Thirty-two records per run, median of seven runs. «([a-z]+),([a-z]+),([a-z]+),([a-z]+),([a-z]+)» «alpha,bravo,charlie,delta,echo» scan::scan<f>.sentinel() 477 ns re2c 554 ns scan::scan<f> 659 ns CTRE 717 ns RE2 15986 ns the same five fields, two hundred characters each scan::scan<f>.sentinel() 57.8 ns re2c 637 ns CTRE 692 ns RE2 11751 ns «first.last@subdomain.example.com» recognition, nothing is extracted scan::match<p>.sentinel().scalar() 436 ns scan::match<p>.scalar() 545 ns re2c 1186 ns RE2 2616 ns CTRE 14339 ns Collapsing groups during parsing
The type informs which group the current character belongs to, and the calculation is performed in place: no loop iteration is saved, and no substring is created. struct tally { unsigned long value = 0; }; struct reading { tally number; std::string_view tail; }; template struct scan::scanner { static constexpr std::string_view pattern() { return R»(\\(((_+)(X|Y)*)*\\))»; } struct state_type { unsigned long total = 0; unsigned place = 0, marks = 0; }; static constexpr state_type begin_groups() { return {}; } static constexpr void opened_group(state_type& one, scan::group_at) { one.place = 0; one.marks = 0; } static constexpr void closed_group(state_type& one, scan::group_at) { unsigned long weight = 1; for (unsigned step = 1; step < one.place; ++step) weight *= 10; one.total += weight * one.marks; } static constexpr void push_group(state_type& one, scan::group_at, char) { ++one.place; } static constexpr void push_group(state_type& one, scan::group_at, char letter) { one.marks += letter == 'Y' ? 2u : 1u; } static constexpr tally finish_groups(state_type one) { return {one.total}; } }; const reading got = scan::scan("value=(__X_XX)abcdefgh").of(); // got.number.value == 12, got.tail == "abcdefgh"
Any call object — allocator, pool, arena, or anything else — can be passed into the invocation that creates a value. The context type is preserved: the scanner state and each of its hooks can be templates for it, meaning that no invocation type is mentioned in the scanner. struct arena { std::pmr::memory_resource* where = nullptr; std::pmr::memory_resource* resource() const { return where; } }; struct numbers { std::pmr::vector values; }; struct both { numbers left; numbers right; }; template struct scan::scanner { static constexpr std::string_view pattern() { return "([0-9]+)(?:,([0-9]+))*"; } template struct state { std::pmr::vector values; int running = 0; }; static state begin_groups() { return {}; } template requires requires(const Told& one) { one.resource(); } static state begin_groups(const Told& told) { return {std::pmr::vector(told.resource()), 0}; } template static void push_group(state& one, scan::group_at, char digit) { one.running = one.running * 10 + (digit - '0'); } template static void closed_group(state& one, scan::group_at) { one.values.push_back(one.running); one.running = 0; } template static numbers finish_groups(state one) { return {std::move(one.values)}; } }; std::pmr::monotonic_buffer_resource bytes; const arena mine{&bytes}; scan::scan(text).of(mine); // one for both places scan::scan(text).of(mine, scan::default_context); // one for each place scan::scan(text).of({mine, scan::default_context}); // same in parentheses const both got = scan::scan(text).with(mine); // The context does not have to be its own type: the allocator gets to the point // where it builds the reading. struct two { std::pmr::string name; std::pmr::string tail; }; const two kept = scan::scan(text).of<( std::pmr::polymorphic_allocator (&bytes));
The hook receives the invocation object itself, not a copy, so the state can store its address: the context lives as long as the call does, and all reading occurs within that call. Other features
- Two layers: the template layer (match, starts_with, search, search_all, split) and the format layer, where "{}" is a field, and the position's value is defined by the fields of the type itself.
- The disambiguation rule is leftmost-first, like in Perl, RE2, and CTRE.
- All of the above works in constant expressions.
- No lookaround, backreferences, and Unicode properties; patterns work with bytes.
- Patterns known only at runtime are not supported.
Building
Compiles with clang and GCC. C++ modules are optional: header files (include/) generated by the demodulizer utility from the same modules are available — each push to main rebuilds, links, and commits them back. The only dependency, Boost.PFR, is not needed when binding packs from C++26 are enabled. FetchContent_Declare(scan GIT_REPOSITORY https://github.com/j4niwzis/scan.git GIT_TAG v0.1.0) FetchContent_MakeAvailable(scan) target_link_libraries(mine PRIVATE scan::scan)
Source: opennet.ru
