[FLINK-40089][table] Implement JSON_LENGTH function by VasShabu · Pull Request #28688 · apache/flink

VasShabu · 2026-07-08T12:33:59Z

What is the purpose of the change

This pull request adds support for the built-in SQL function JSON_LENGTH, which returns the number of top-level elements in a JSON array or object. Scalars have a length of 1, and NULL input returns NULL. Nested arrays or objects are not counted recursively.

Brief change log

Added JsonLengthFunction implementing the JSON_LENGTH built-in function
Registered JSON_LENGTH as a built-in function in the function catalog
Added test cases for JSON_LENGTH in JsonFunctionsITCase

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

mvn test -pl flink-table/flink-table-runtime -Dtest=JsonFunctionsITCase
Added test cases for JSON_LENGTH covering scalars, arrays, objects, and NULL input in JsonFunctionsITCase

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs (sql_functions.yml)

Was generative AI tooling used to co-author this PR?

Yes (Claude)

flinkbot · 2026-07-08T12:43:54Z

CI report:

a9facf2 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

raminqaf

Thanks for the contribution @VasShabu! Left some comments!

raminqaf · 2026-07-08T12:47:39Z

+    public static Integer jsonLength(String input, String pathSpec) {
+        return jsonLength(jsonApiCommonSyntax(input, pathSpec));
+    }
+
+    private static Integer jsonLength(JsonPathContext context) {
+        if (context.hasException()) {
+            return null;
+        }
+        final Object value = context.obj;
+        if (value == null) {
+            return null;
+        }
+        if (value instanceof Map) {
+            return ((Map<?, ?>) value).size();
+        }
+        if (value instanceof Collection) {
+            return ((Collection<?>) value).size();
+        }
+        return 1;
+    }


Why to we need this? Don't we have already the runtime class JsonLengthFunction?

raminqaf · 2026-07-08T12:48:56Z

+        super(BuiltInFunctionDefinitions.JSON_LENGTH, context);
+    }
+
+    public @Nullable Integer eval(@Nullable StringData jsonInput) {


nit: here and everywhere else make parameters/variables that are immutable final (not enforced on Flink but nicer to read).

Suggested change

public @Nullable Integer eval(@Nullable StringData jsonInput) {

public @Nullable Integer eval(final @Nullable StringData jsonInput) {

gustavodemorais

Thanks for the contribution, @VasShabu! Good job on the first version of the PR. We'll go through some iterations to make sure things look good

Apart from what we already flagged above: we have two implementations of the same function. eval(json) uses a new Jackson readTree, eval(json, path) uses the existing Calcite path in SqlJsonUtils. Different parsers + different data models = they can disagree on the same input. We can make the design more consistent by using only one path. Duplicate ObjectMapper - reuse the configured one in SqlJsonUtils instead of a second bare instance. This is important because built-in functions are on the hot path. The logic might be invoked millions of times in a couple of seconds and we have to make sure they allocate as little as possible and their performance are optimized or else the engine will be wasting resources.

That said, In this case, we could go two paths

Best performance: We build a custom parser to optimize it for length checking by skipping the children. MAPPER.getFactory().createParser(str), read first token; if START_ARRAY/START_OBJECT walk top level counting, skipping children via parser.skipChildren(). No tree alloc. Best for length-only.
Simplest reuse the shared SqlJsonUtils path so both overloads share one parser.

Take a look at both paths and let me know what you think

gustavodemorais · 2026-07-08T14:28:37Z

+        } else if (jsonNode.isTextual()
+                || jsonNode.isNumber()
+                || jsonNode.isBoolean()
+                || jsonNode.isBinary()) {


Isn't this check dead code? Can a jsonNode be binary? You can try to write a test to check if it's dead code or it's possible

snuyanzin

Are we sure the selected approach is the right one?
With this approach the same JSON will be parsed several times for queries like

SELECT JSON_LENGTH(my_json), JSON_LENGTH(my_json), JSON_LENGTH(my_json) FROM...

or

SELECT JSON_LENGTH(my_json), JSON_VALUE(my_json, '$.a') FROM...

snuyanzin · 2026-07-08T18:21:32Z

+    table: jsonLength(jsonObject[, path])
+    description: |
+      Returns the number of elements in a JSON document, or the length of the value at the specified path if one is provided. 
+      Returns NULL if the argument is NULL or the path does not locate a value.


this does not look true
simple tests show different behavior

SELECT json_length('$'); SELECT json_length('{');

each returns -1 and the doc says nothing about this or did I miss anything?

snuyanzin

More findings:

Doc doesn't reflect the real behavior: it returns -1 in case of invalid json, while docs says about NULL, e.g. query SELECT JSON_LENGTH('null');
Strange handling of null values
like a query
```
SELECT json_length('{"a":[true, false, null]}', '$.a[1]')
```
both in MySQL and in Flink it returns 1
now
```
SELECT json_length('{"a":[true, false, null]}', '$.a[2]')
```
in MySQL it returns 1, in Flink it returns -1 why?

Seems need to check what the behavior should be in case of lax/strict since right now it is weird

SELECT json_length('{"x": {"a":1}, "y": {"a":[2, 3]}}', 'lax $.*.a');  -- returns 2

SELECT json_length('{"x": {"a":[1, 2, 3, 4]}, "y": {"a":[2, 3]}}', 'lax $.*.a');  -- returns 2

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a'); -- returns 3

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[0]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[1]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[2]'); --returns 1

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}, "z":{"a":1}}', 'lax *.a[3]');  --returns 0

davidradl · 2026-07-13T13:57:17Z

        """
        return _unary_op("jsonUnquote")(self)

+    def json_length(self, path = None) -> 'Expression':


It seems that many databases throw an error, I am curious why we are not throwing an error. Also I see that there are best practises that we should document like using IF(string_column IS JSON, to protect against non json and avoid potential errors.

…single parse

snuyanzin · 2026-07-14T09:26:45Z

+        final JsonPathContext context = jsonApiCommonSyntax(parsedInput, pathSpec);
+        final Object value = context.hasException() ? null : context.obj;
+
+        if (value instanceof LinkedList) {


is there a reason we expect LinkedList instead of just List?

the comment marked as resolved however no answer yet
can you elaborate here?

snuyanzin · 2026-07-14T09:28:13Z

+logger.codelog.name = org.apache.flink.table.runtime.generated.CompileUtils
+logger.codelog.level = DEBUG
+logger.codelog.appenderRef.test.ref = TestLogger


it might be helpful for local debug, however why do we need in repo and making it executable for every ci run?

raminqaf

Another pass

raminqaf · 2026-07-14T10:08:01Z

+            if (matched.size() != 1) {
+                if (context.mode == PathMode.STRICT || matched.isEmpty()) {
+                    return null;
+                }
+                return matched.size();
+            }


Could you please elaborate on this? In which conditions we get a null and in which conditions we get the size of the matched?

looks like not addressed yet

snuyanzin · 2026-07-14T16:21:49Z

+#logger.codelog.name = org.apache.flink.table.runtime.generated.CompileUtils
+#logger.codelog.level = DEBUG
+#logger.codelog.appenderRef.test.ref = TestLogger
+


can we just drop it?

keeping log setting for every helper here might be overkill

snuyanzin · 2026-07-14T16:24:17Z

+            generateCallWithStmtIfArgsNotNull(ctx, resultType, operands, resultNullable = true) {
+              argTerms =>
+                val inputTerm = s"${argTerms.head}.toString()"
+                val (varName, parseCode) =
+                  ctx.getReusableInputUnboxingExprs(inputTerm, Int.MinValue) match {
+                    case Some(expr) => (expr.resultTerm, "")
+                    case None =>
+                      val v = CodeGenUtils.newName(ctx, "jsonParsed")
+                      ctx.addReusableMember(
+                        s"${classOf[SqlJsonUtils.JsonValueContext].getName} $v;")
+                      ctx.addReusableInputUnboxingExprs(
+                        inputTerm,
+                        Int.MinValue,
+                        GeneratedExpression(v, "false", "", null))
+                      (v, s"$v = ${qualifyMethod(BuiltInMethods.JSON_PARSE)}($inputTerm);")
+                  }
+                val (method, terms) =
+                  if (operands.length > 1)
+                    (BuiltInMethods.JSON_LENGTH_PATH, Seq(varName, s"${argTerms(1)}.toString()"))
+                  else
+                    (BuiltInMethods.JSON_LENGTH, Seq(varName))
+                (parseCode, s"${qualifyMethod(method)}(${terms.mkString(", ")})")


is it possible to extract it ideally to java helper?

snuyanzin

it seems the lax problem is still not solved

to highlight the issue there are 2 queries:
1.

SELECT json_length('{"x": {"a":[3, 2 , 3]}}', 'lax *.a');

which returns 3, I guess the size of x.a.
2.

SELECT json_length('{"x": {"a":[3, 2 , 3]}, "y": {"a":null}}', 'lax *.a');

now returns 2 which is amount of x.a, not the amount of elements.

There is nothing about this in doc, and it is very non obvious that depending on query the same syntax will mean completely different things

and there is no info about this in docs

[FLINK-40089][table] Implement JSON_LENGTH function

200b15a

raminqaf reviewed Jul 8, 2026

View reviewed changes

gustavodemorais reviewed Jul 8, 2026

View reviewed changes

snuyanzin requested changes Jul 8, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 8, 2026

snuyanzin reviewed Jul 8, 2026

View reviewed changes

snuyanzin requested changes Jul 8, 2026

View reviewed changes

davidradl reviewed Jul 13, 2026

View reviewed changes

[FLINK-40089][table] Fix JSON_LENGTH null handling, used codegen for …

9f85f67

…single parse