re2c 1.2

On Friday, August 2nd, the release of re2c was announced — a free lexical analyzer generator for C and C++ languages. Recall that re2c was developed in 1993 by Peter Bambulis as an experimental generator of very fast lexical analyzers, distinguished from other generators by the speed of the generated code and an unusually flexible user interface that allows for easy and efficient integration of analyzers into existing codebases. Since then, the project has been developed by the community and continues to serve as a platform for experiments and research in formal grammars and finite automata.

Key innovations in version 1.2:

• A new (simplified) way to check for end of input
  (English: "EOF rule").
  For this, the configuration re2c:eof has been added,
  allowing the selection of a terminal symbol,
  and a special rule $ that triggers if the lexer
  successfully reaches the end of the input data.
  Historically, re2c offers several methods for checking for
  end of input, varying in limitations, efficiency, and simplicity
  of use. The new method is designed to simplify coding while
  remaining efficient and widely applicable. The old methods
  still work and may be preferable in certain cases.

• The ability to include external files using the directive has been added,
  /*!include:re2c "file.re" */, где file.re
  this is the name of the included file. Re2c searches for files in the directory of the including file,
  as well as in the list of paths specified using the -I option.
  Included files may include other files.
  Re2c provides "standard" files in the include/ directory of the project — it is assumed that useful definitions
  of regular expressions will accumulate there, something akin to a standard library.
  So far, at the request of users, one file with definitions of Unicode categories has been added.
  The ability to generate header files with arbitrary

• content using the -t —type-header options (or corresponding
  configurations) and new directives /*!header:re2c:on*/ has been introduced,
  when re2c should generate definitions of variables, structures, and macros,
  /*!header:re2c:off*/. Это может быть полезно в случаях,
  used in other translation units.
  Re2c now understands UTF8 literals and character classes in regular expressions.

• By default, re2c parses expressions like "∀x ∃y" as.
  By default, re2c parses expressions like "∀x ∃y" as.
  A sequence of 1-bit ASCII characters e2 88 80 78 20 e2 88 83 79
  (hex codes), and users have to manually escape Unicode characters:
  "u2200x u2203y". This is very inconvenient and unexpected for many
  users (as evidenced by the constant bug reports). Therefore, now
  re2c provides the option —input-encoding ,
  which allows changing the behavior and parsing "∀x ∃y" as
  2200 78 20 2203 79.

• Re2c now allows using regular re2c blocks in -r —reuse mode.
  This is convenient if the input file contains many blocks, and only some of them
  need to be reused.

• A format for warnings and error messages can now be set
  using the new option —location-format . The GNU format displays
  as filename:line:column:, and the MSVC format — as filename(line,column).
  This feature may be useful for IDE enthusiasts.
  An option —verbose has also been added, which outputs a brief success message.

• The compatibility mode with flex has been improved—some parsing errors and
  incorrect operator precedence in rare cases have been fixed.
  Historically, the option -F —flex-support allows writing code
  in a mix of flex style and re2c style, which complicates parsing somewhat.
  The compatibility mode with flex is rarely used in new code,
  but re2c continues to support it for backward compatibility.

• The subtraction operator for character classes / is now applied
  before expanding the encoding, which allows it to be applied in more cases,
  if a variable-length character encoding (such as UTF8) is used.

• The output file is now created atomically: re2c first creates a temporary file
  and writes the result to it, then renames the temporary file to the output
  in one operation.

• The documentation has been completed and rewritten; in particular, new
  chapters have been added on buffer filling
  and on methods to check for the end of input data.
  The new documentation is compiled as
  a comprehensive single-page manual
  with examples (the same sources are rendered in manpage and online documentation).
  Weak attempts have been made to improve the site's readability on phones.

• From a developer's perspective, re2c has gained a more complete debugging subsystem.
  Debugging code is now disabled in release builds and
  can be enabled using the configure option —enable-debug.

This release took a long time — almost a whole year.
Most of the time, as always, was spent on developing the theoretical foundation and writing
article «Efficient POSIX Submatch Extraction on NFA».
The algorithms described in the article are implemented in the experimental library libre2c
(building the library and benchmarks is disabled by default and can be enabled with the configure option
—enable-libs). The library is not intended to compete with existing
projects like RE2, but rather as a research platform for developing new
algorithms (which can later be used in re2c or other projects).
It is also convenient for testing, benchmarking, and creating bindings to other languages.

Thanks from the re2c developers to everyone who helped make this release happen,
and to the community in general for ideas, bug reports, patches, morale, etc. ;]

Source: linux.org.ru