From flex and bison to cparse

mstring started in 2006 as a flex scanner and a bison grammar. For 1.1 the front end was rewritten on cparse — and writing a parser by hand, rule by rule, turned out to be the best code review the old one ever had.

Why rewrite a working parser

It was not quite working. The scanner declared its start conditions with %s — inclusive — so every rule without a condition was active inside strings, include names and macros too. Most of the time the longest-match rule happened to win; sometimes it did not. Two more build-time tools for one small grammar was the other reason.

What the rewrite found

Reading each rule to reimplement it showed what the old grammar actually accepted:

• $CHARSET UTF-8 was a syntax error — the - was not allowed in a name.
• A # comment after a text line swallowed the newline the grammar was waiting for.
• \\ in a message became a lone backslash in the generated C++ literal — where it escaped the next character.
• MEMBER AS functions were defined but never declared, so they never compiled; $SYSLOG MEMBER OF silently produced a free function.
• The generator built a std::locale for every language to compare names — and threw if the build machine did not have that locale installed.
• The exception classes used throw ( ) specifications, which C++20 removed.

The full list is in the changelog of 1.1.

The new parser

MStringParser derives from cparse's Parser. The language is line oriented, so the grammar is a loop: skip empty lines, then either a directive or a line of the current message. Each construct is one method — directive(), parameterLine(), quotedTextLine(), macro() — and reports errors with the cursor's line and column:

app.msg:3:1: error: parameters must be declared before the message texts

An $IMPORT runs a second parser instance on the imported file, sharing the state. The generated code is checked the only way that counts: the tests compile it, with -Werror, and run it.

The grammar, drawn as railroad diagrams, is on the Language pages.