C++ regex parse tree and RegexTreeViewSig implementation (Phase 1 of ReDoS support)#22200
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| */ | ||
|
|
||
| import cpp | ||
| private import semmle.code.cpp.dataflow.new.DataFlow |
| private import cpp | ||
| private import semmle.code.cpp.regex.RegexTreeView::RegexTreeView as TreeView | ||
| import codeql.regex.nfa.SuperlinearBackTracking::Make<TreeView> as SuperlinearBackTracking | ||
| private import semmle.code.cpp.ir.dataflow.DataFlow |
|
|
||
| // Instantiate the shared analysis modules with our RegexTreeView. | ||
| // If RegexTreeView does not satisfy RegexTreeViewSig, these lines cause a compile error. | ||
| private module TestSuperlinear = SuperlinearBackTracking::Make<RegexTreeView>; |
| // If RegexTreeView does not satisfy RegexTreeViewSig, these lines cause a compile error. | ||
| private module TestSuperlinear = SuperlinearBackTracking::Make<RegexTreeView>; | ||
|
|
||
| private module TestNfaUtils = NfaUtils::Make<RegexTreeView>; |
|
QHelp previews: cpp/ql/src/Security/CWE/CWE-1333/PolynomialReDoS.qhelpPolynomial regular expression used on uncontrolled dataSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, and potentially allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The C++ standard library regular expression engine ( Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time complexity does not matter. ExampleConsider this use of a regular expression, which removes all leading and trailing whitespace in a string: std::string trim(const std::string &text) {
static const std::regex re("^\\s+|\\s+$");
return std::regex_replace(text, re, ""); // BAD
}
The sub-expression This ultimately means that the time cost of trimming a string is quadratic in the length of the string. So a string like Avoid this problem by rewriting the regular expression to not contain the ambiguity about when to start matching whitespace sequences. For instance, by using a negative look-behind ( Note that the sub-expression ExampleAs a similar, but slightly subtler problem, consider the regular expression that matches lines with numbers, possibly written using scientific notation: static const std::regex numRe("^0\\.\\d+E?\\d+$");
if (std::regex_search(input, numRe)) { /* ... */ } // BAD
The problem with this regular expression is in the sub-expression This is problematic for strings that do not end with a digit. Such a string will force the regular expression engine to process each digit sequence once per digit in the sequence, again leading to a quadratic time complexity. To make the processing faster, the regular expression should be rewritten such that the two ExampleSometimes it is unclear how a regular expression can be rewritten to avoid the problem. In such cases, it often suffices to limit the length of the input string. For instance, the following regular expression is used to match numbers, and on some non-number inputs it can have quadratic time complexity: static const std::regex numRe(
"^(\\+|-)?(\\d+|(\\d*\\.\\d*))?(E|e)?([-+])?(\\d+)?$");
if (std::regex_match(input, numRe)) { /* ... */ }
It is not immediately obvious how to rewrite this regular expression to avoid the problem. However, you can mitigate performance issues by limiting the length of the input to a small constant, which will always finish in a reasonable amount of time: if (input.size() > 1000) {
throw std::invalid_argument("Input too long");
}
if (std::regex_match(input, numRe)) { /* ... */ }
References
cpp/ql/src/Security/CWE/CWE-1333/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, and potentially allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The C++ standard library regular expression engine ( Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time complexity does not matter. ExampleConsider this regular expression: static const std::regex re("^(a+)+$");
if (std::regex_match(input, re)) { /* ... */ } // BAD
Its sub-expression This problem can be avoided by rewriting the regular expression so that the repetition of the outer group cannot overlap with the repetition of the inner group. For example, by simply using a single repetition: static const std::regex re("^a+$");
if (std::regex_match(input, re)) { /* ... */ }
ExampleAs a slightly subtler example, consider the regular expression static const std::regex re("^(a|a)*$");
if (std::regex_match(input, re)) { /* ... */ } // BAD
This can be fixed by removing the ambiguity between the two branches of the alternative, for instance by rewriting the pattern as References
java/ql/src/Security/CWE/CWE-730/PolynomialReDoS.qhelpPolynomial regular expression used on uncontrolled dataSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine provided by Java uses a backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form Note that Java versions 9 and above have some mitigations against ReDoS; however they aren't perfect and more complex regular expressions can still be affected by this problem. RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. Alternatively, an alternate regex library that guarantees linear time execution, such as Google's RE2J, may be used. ExampleConsider this use of a regular expression, which removes all leading and trailing whitespace in a string: Pattern.compile("^\\s+|\\s+$").matcher(text).replaceAll("") // BADThe sub-expression This ultimately means that the time cost of trimming a string is quadratic in the length of the string. So a string like Avoid this problem by rewriting the regular expression to not contain the ambiguity about when to start matching whitespace sequences. For instance, by using a negative look-behind ( Note that the sub-expression ExampleAs a similar, but slightly subtler problem, consider the regular expression that matches lines with numbers, possibly written using scientific notation: "^0\\.\\d+E?\\d+$"" The problem with this regular expression is in the sub-expression This is problematic for strings that do not end with a digit. Such a string will force the regular expression engine to process each digit sequence once per digit in the sequence, again leading to a quadratic time complexity. To make the processing faster, the regular expression should be rewritten such that the two ExampleSometimes it is unclear how a regular expression can be rewritten to avoid the problem. In such cases, it often suffices to limit the length of the input string. For instance, the following regular expression is used to match numbers, and on some non-number inputs it can have quadratic time complexity: Pattern.matches("^(\\+|-)?(\\d+|(\\d*\\.\\d*))?(E|e)?([-+])?(\\d+)?$", str); It is not immediately obvious how to rewrite this regular expression to avoid the problem. However, you can mitigate performance issues by limiting the length to 1000 characters, which will always finish in a reasonable amount of time. if (str.length() > 1000) {
throw new IllegalArgumentException("Input too long");
}
Pattern.matches("^(\\+|-)?(\\d+|(\\d*\\.\\d*))?(E|e)?([-+])?(\\d+)?$", str); References
java/ql/src/Security/CWE/CWE-730/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine provided by Java uses a backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form Note that Java versions 9 and above have some mitigations against ReDoS; however they aren't perfect and more complex regular expressions can still be affected by this problem. RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. Alternatively, an alternate regex library that guarantees linear time execution, such as Google's RE2J, may be used. ExampleConsider this regular expression: ^_(__|.)+_$Its sub-expression This problem can be avoided by rewriting the regular expression to remove the ambiguity between the two branches of the alternative inside the repetition: ^_(__|[^_])+_$References
javascript/ql/src/Performance/PolynomialReDoS.qhelpPolynomial regular expression used on uncontrolled dataSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engines provided by many popular JavaScript platforms use backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this use of a regular expression, which removes all leading and trailing whitespace in a string: text.replace(/^\s+|\s+$/g, ''); // BADThe sub-expression This ultimately means that the time cost of trimming a string is quadratic in the length of the string. So a string like Avoid this problem by rewriting the regular expression to not contain the ambiguity about when to start matching whitespace sequences. For instance, by using a negative look-behind ( Note that the sub-expression ExampleAs a similar, but slightly subtler problem, consider the regular expression that matches lines with numbers, possibly written using scientific notation: /^0\.\d+E?\d+$/.test(str) // BADThe problem with this regular expression is in the sub-expression This is problematic for strings that do not end with a digit. Such a string will force the regular expression engine to process each digit sequence once per digit in the sequence, again leading to a quadratic time complexity. To make the processing faster, the regular expression should be rewritten such that the two ExampleSometimes it is unclear how a regular expression can be rewritten to avoid the problem. In such cases, it often suffices to limit the length of the input string. For instance, the following regular expression is used to match numbers, and on some non-number inputs it can have quadratic time complexity: /^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$/.test(str) // BADIt is not immediately obvious how to rewrite this regular expression to avoid the problem. However, you can mitigate performance issues by limiting the length to 1000 characters, which will always finish in a reasonable amount of time. if (str.length > 1000) {
throw new Error("Input too long");
}
/^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$/.test(str)References
javascript/ql/src/Performance/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engines provided by many popular JavaScript platforms use backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this regular expression: /^_(__|.)+_$/Its sub-expression This problem can be avoided by rewriting the regular expression to remove the ambiguity between the two branches of the alternative inside the repetition: /^_(__|[^_])+_$/References
python/ql/src/Security/CWE-730/PolynomialReDoS.qhelpPolynomial regular expression used on uncontrolled dataSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine provided by Python uses a backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this use of a regular expression, which removes all leading and trailing whitespace in a string: re.sub(r"^\s+|\s+$", "", text) # BADThe sub-expression This ultimately means that the time cost of trimming a string is quadratic in the length of the string. So a string like Avoid this problem by rewriting the regular expression to not contain the ambiguity about when to start matching whitespace sequences. For instance, by using a negative look-behind ( Note that the sub-expression ExampleAs a similar, but slightly subtler problem, consider the regular expression that matches lines with numbers, possibly written using scientific notation: ^0\.\d+E?\d+$ # BADThe problem with this regular expression is in the sub-expression This is problematic for strings that do not end with a digit. Such a string will force the regular expression engine to process each digit sequence once per digit in the sequence, again leading to a quadratic time complexity. To make the processing faster, the regular expression should be rewritten such that the two ExampleSometimes it is unclear how a regular expression can be rewritten to avoid the problem. In such cases, it often suffices to limit the length of the input string. For instance, the following regular expression is used to match numbers, and on some non-number inputs it can have quadratic time complexity: match = re.search(r'^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$', str) It is not immediately obvious how to rewrite this regular expression to avoid the problem. However, you can mitigate performance issues by limiting the length to 1000 characters, which will always finish in a reasonable amount of time. if len(str) > 1000:
raise ValueError("Input too long")
match = re.search(r'^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$', str) References
python/ql/src/Security/CWE-730/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine provided by Python uses a backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this regular expression: ^_(__|.)+_$Its sub-expression This problem can be avoided by rewriting the regular expression to remove the ambiguity between the two branches of the alternative inside the repetition: ^_(__|[^_])+_$References
ruby/ql/src/queries/security/cwe-1333/PolynomialReDoS.qhelpPolynomial regular expression used on uncontrolled dataSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine used by the Ruby interpreter (MRI) uses backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Note that Ruby 3.2 and later have implemented a caching mechanism that completely eliminates the worst-case time complexity for the regular expressions flagged by this query. The regular expressions flagged by this query are therefore only problematic for Ruby versions prior to 3.2. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this use of a regular expression, which removes all leading and trailing whitespace in a string: text.gsub!(/^\s+|\s+$/, '') # BADThe sub-expression This ultimately means that the time cost of trimming a string is quadratic in the length of the string. So a string like Avoid this problem by rewriting the regular expression to not contain the ambiguity about when to start matching whitespace sequences. For instance, by using a negative look-behind ( Note that the sub-expression ExampleAs a similar, but slightly subtler problem, consider the regular expression that matches lines with numbers, possibly written using scientific notation: /^0\.\d+E?\d+$/ # BADThe problem with this regular expression is in the sub-expression This is problematic for strings that do not end with a digit. Such a string will force the regular expression engine to process each digit sequence once per digit in the sequence, again leading to a quadratic time complexity. To make the processing faster, the regular expression should be rewritten such that the two ExampleSometimes it is unclear how a regular expression can be rewritten to avoid the problem. In such cases, it often suffices to limit the length of the input string. For instance, the following regular expression is used to match numbers, and on some non-number inputs it can have quadratic time complexity: is_matching = /^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$/.match?(str)It is not immediately obvious how to rewrite this regular expression to avoid the problem. However, you can mitigate performance issues by limiting the length to 1000 characters, which will always finish in a reasonable amount of time. if str.length > 1000
raise ArgumentError, "Input too long"
end
is_matching = /^(\+|-)?(\d+|(\d*\.\d*))?(E|e)?([-+])?(\d+)?$/.match?(str)References
ruby/ql/src/queries/security/cwe-1333/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, or even allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine used by the Ruby interpreter (MRI) uses backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or even exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Note that Ruby 3.2 and later have implemented a caching mechanism that completely eliminates the worst-case time complexity for the regular expressions flagged by this query. The regular expressions flagged by this query are therefore only problematic for Ruby versions prior to 3.2. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time-complexity does not matter. ExampleConsider this regular expression: /^_(__|.)+_$/Its sub-expression This problem can be avoided by rewriting the regular expression to remove the ambiguity between the two branches of the alternative inside the repetition: /^_(__|[^_])+_$/References
swift/ql/src/queries/Security/CWE-1333/ReDoS.qhelpInefficient regular expressionSome regular expressions take a long time to match certain input strings to the point where the time it takes to match a string of length n is proportional to nk or even 2n. Such regular expressions can negatively affect performance, and potentially allow a malicious user to perform a Denial of Service ("DoS") attack by crafting an expensive input string for the regular expression to match. The regular expression engine used by Swift uses backtracking non-deterministic finite automata to implement regular expression matching. While this approach is space-efficient and allows supporting advanced features like capture groups, it is not time-efficient in general. The worst-case time complexity of such an automaton can be polynomial or exponential, meaning that for strings of a certain shape, increasing the input length by ten characters may make the automaton about 1000 times slower. Typically, a regular expression is affected by this problem if it contains a repetition of the form RecommendationModify the regular expression to remove the ambiguity, or ensure that the strings matched with the regular expression are short enough that the time complexity does not matter. ExampleConsider the following regular expression: Its sub-expression This problem can be avoided by rewriting the regular expression to remove the ambiguity between the two branches of the alternative inside the repetition: References
|
…ts in regex files
The two "Dead code" alerts (100379, 100380) point at the isSource and isSink predicates in `private module RegexPatternFlowConfig implements DataFlow::ConfigSig` in cpp/ql/lib/semmle/code/cpp/regex/RegexFlowConfigs.qll. Rationale (Case C): The predicates are genuine and correctly wired - `RegexPatternFlowConfig` is consumed by `TaintTracking::Global<RegexPatternFlowConfig>` on the very next line, which drives the `usedAsRegex` / `regexMatchedAgainst` public predicates. The DeadCode query cannot see through signature-parameterized modules and therefore flags every implementation-side `isSource`/`isSink` inside a private ConfigSig module (the same false positive is present on many existing configs across the codebase). Removing the leftover `private` qualifier on the config module resolves both alerts by making the module publicly reachable; the taint-tracking alias below it remains private, so no additional flow surface is exposed to consumers of this library.
This reverts commit cc95e25.
C++ (
codeql/cpp-all) has no regex parse tree and noRegexTreeViewSigimplementation, blocking instantiation of the sharedcodeql/regexReDoS engines. This PR adds the self-contained foundation — parser + tree view — with no queries and no dataflow (Phase 2).New:
ParseRegExp.qllcpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qllclass RegExp extends StringLiteralproviding offset-span predicates for the full ECMAScript construct set:*,+,?,{n},{n,m},{n,}(greedy and lazy)(...),(?:...),(?<name>...),(?=...),(?!...),(?<=...),(?<!...).,^,$,\b,\B)\1and ECMAScript named\k<name>\n,\r,\t,\xhh,\uhhhh,\u{...}, identity escapesECMAScript semantics throughout (targeting
std::regex's default mode). Phase 1 conservatively treats allStringLiterals as candidate regexes; Phase 2 will restrict viastd::regexconstruction-site dataflow.New:
RegexTreeView.qllcpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qllprivate newtype TRegExpParentwith 9 variants +module Impl implements RegexTreeViewSig:RegExpQuantifier/RegExpStar/RegExpPlus/RegExpOpt/RegExpRange,RegExpAlt,RegExpSequence,RegExpGroup,RegExpCharacterClass/RegExpCharacterRange,RegExpNormalChar/RegExpEscape/RegExpCharacterClassEscape,RegExpSpecialChar,RegExpDot/RegExpAnchor/RegExpCaret/RegExpDollar/RegExpWordBoundary/RegExpNonWordBoundary,RegExpSubPattern/lookahead/lookbehind variants,RegExpBackRef,RegExpConstantisPossessive → none(),matchesAnyPrefix/AnySuffix → any(),isIgnoreCase/isDotAll → none()(conservative; TODO for Phase 2 construction-site flag detection)hasLocationInfomaps term offsets back to source columns within the enclosingStringLiteralimport Impl as RegexTreeView(functor argument) andimport Impl(direct use)Compile check
cpp/ql/test/library-tests/regex/CompileCheck.qlinstantiates bothSuperlinearBackTracking::Make<RegexTreeView>andNfaUtils::Make<RegexTreeView>to verify the signature is satisfied end-to-end.Tests
cpp/ql/test/library-tests/regex/— test source covering quantifiers, nested quantifiers (relevant for ReDoS:(a+)+), named groups, lookahead/lookbehind, backreferences, anchors, character classes; plus parse-tree printer (parse.ql), table query (regexp.ql), andfailedToParseconsistency check (Consistency.ql).Packaging
cpp/ql/lib/qlpack.yml— addscodeql/regex: ${workspace}dependency.