REGEX Dialect

[mzuenni]

It is currently not specified what regular expression dialct is used, and this has already been changed in the past: fc72292

Note that this has many implications:

• \d vs [:digit:]
• longest vs first matching: REGEX("p|ps") RGEX("s") EOF will either accept ps or not
• a lot of common features are not present in ERE

What was the reason to switch from ECMAScript to ERE here? (ECMAScript would be the more natural choice IMO)

[eldering]

Not sure exactly anymore. Looking at my old commit message it seems it was documented somewhere.

Regex dialects is a bit of a messy business, so it might be that I chose ERE as I think it should be one of the most well-defined ones.

[mzuenni]

Regex dialects is a bit of a messy business

yeah...

I chose ERE as I think it should be one of the most well-defined ones

It seems like it is one of the dialect that is not properly supported by any python package ):

[eldering]

TBH, I also think that if as a CTD program writer you need to care about such levels of detail such as greediness of matching, then you're pushing (beyond) the limits of checktestdata as a tool.

I wonder if we could in a reasonable way specify a somewhat smaller subset of regex that can be implemented in basically any programming language because it's a subset (except for nitty gritty greediness etc.) of a regex dialect that that language has support for.

[mzuenni]

TBH, I also think that if as a CTD program writer you need to care about such levels of detail such as greediness of matching, then you're pushing (beyond) the limits of checktestdata as a tool.

That is true, but there is also nothing that stops/warns you currently if you rely on the specific current behaviour.

[mzuenni]

I wonder if we could in a reasonable way specify a somewhat smaller subset of regex that can be implemented in basically any programming language because it's a subset

I would guess this is a reasonable subset:

• .: matches anything (including newline)
• []: character class, (allowing - to specify ranges and ^ at the start to invert the character class)
• {m,n}: match the previous regex between m and n times (m and n inclusive). if m is empty it is 0, if n is empty it is ∞.
• {m}: shorthand for {m,m}
• *: shorthand for {0,}
• +: shorthand for {1,}
• ?: shorthand for {0,1}
• |: combines two regex with or
• (): parenthesis to limit the scope of |
• \: escape

This would not allow some common features:

• \d etc.: commonly used character classes (not currently supported by checktestdata)
• [:digit:] etc: posix style classes like (we should warn if a character class contains a symbol twice because currently checktestdata recognizes these)
• *?, +?, ??, *+, ++, ?+: minimal and posessive quantifiers (probably not needed, not sure if supported)
• () groups (probably not needed, not sure if supported)
• ^ and $: to match the start and end of a line (i dont think that is useful for checktestdata)

[mzuenni]

Additionally I would specify leftmost-first matching instead of leftmost-longest. It seems that almost every regex implementation does leftmost-first

[mzuenni]

@eldering I added a more careful writeup here as well as a python reference implementation here