Add ServiceLoader-based discovery of TransformFunction implementations

[xiangfu0]

Summary

Adds automatic ServiceLoader-based discovery of external block-oriented TransformFunction implementations, so an application or plugin JAR can register transform functions without setting pinot.server.transforms or making application-specific server startup changes. The discovery mechanism itself is hosted as a reusable pinot-spi helper — PluginManager#loadServiceProviders(Class) — which also replaces the hand-rolled ServiceLoader enumeration previously duplicated in PinotRuleSet and OperatorTypeRegistry.

(This PR consolidates the previously separate #19262 into a single change, since the helper extraction and the discovery feature are easiest to review together.)

Unlike @ScalarFunctions, external TransformFunction implementations previously could not self-register: TransformFunctionFactory only knew built-ins plus classes explicitly listed in pinot.server.transforms. Converting such functions to scalar functions is not an option when they need ValueBlock-level execution and batching.

How it works

An extension JAR ships a standard service descriptor:

META-INF/services/org.apache.pinot.core.operator.transform.function.TransformFunction

listing one implementation class per line, e.g.:

com.example.AiFilterTransformFunction
com.example.AiClassifyTransformFunction
com.example.AiGenerateTransformFunction

TransformFunctionFactory.init(...) (called by ServerInstance after plugins are loaded, before query traffic) discovers providers via the new PluginManager#loadServiceProviders(Class) from the thread context classloader (the application classpath in a standard deployment) and from every classloader returned by PluginManager.get().getPluginClassLoaders(). Discovery never runs on the query path; get() performs plain lookups against an immutable snapshot.

Provider contract

A provider must be a public concrete class implementing TransformFunction, with a public no-argument constructor, returning a non-null/non-blank name from getName(). The startup-created instance is used only to obtain and validate the name and implementation class; it is never init()-ed or evaluated — query execution keeps constructing fresh instances through the factory.

Registration and collision rules

• Built-in registrations are always preserved; names are canonicalized with TransformFunctionFactory.canonicalize().
• The same implementation visible through overlapping application/plugin classloaders is de-duplicated (first sighting wins; a same-named class from a different classloader indicates version skew and is skipped with a WARN); re-registering the identical implementation is a no-op.
• Server startup fails (original cause preserved) if two different discovered classes claim the same canonical name, a discovered class collides with a built-in, a service descriptor is malformed, a provider cannot be constructed, or a provider returns a null/blank name. Collision errors include the function name, both implementation classes, and their classloaders.
• pinot.server.transforms is preserved for backward compatibility and is applied after discovery, retaining its historical explicit-override semantics. Discovery skips classes that are also explicitly configured, so shipping a descriptor for an explicitly configured class (e.g. one that overrides a built-in name) never causes a collision failure.
• A discovered name that matches an existing scalar function is registered with a WARN (not a failure): providing a block-oriented implementation of one's own scalar function is the same pattern the built-ins use, but the log flags potential semantic divergence (literal-only invocations still constant-fold through the scalar implementation at compile time).

Initialization and thread safety

init(...) is synchronized, builds and validates the registry in a local map, and atomically publishes an immutable snapshot to a volatile field. Readers never observe a partially initialized registry; repeated initialization is idempotent; a failed init leaves the previously published registry in place. There is no per-query or per-block discovery overhead.

New public SPI surface (reviewer attention)

PluginManager#loadServiceProviders(Class) plus the nested immutable PluginManager.ServiceProvider<S> pair type (provider instance + human-readable classloader-source description) are a permanent additive pinot-spi contract: context-classloader-first discovery order, first-sighting-wins dedup by class name, WARN-and-skip on classloader version skew, an unmodifiable freshly computed result list, and fail-fast IllegalStateException wrapping of ServiceConfigurationError with source context. Per-provider validation and registration policy stay with callers. No existing signature changed; getPluginClassLoaders() remains for callers needing the raw classloaders.

The two prior hand-rolled copies of this enumeration were migrated to the helper:

• PinotRuleSet.loadFromServiceLoader() (pinot-query-planner) — customizer ordering preserved (context-classloader defaults before plugin customizers).
• OperatorTypeRegistry static initializer (pinot-query-runtime) — PLUGIN_ID_FLOOR and duplicate-id checks preserved.

Deliberate drift at those two call sites (unified onto the fail-fast contract): a malformed descriptor or failing provider constructor now surfaces as IllegalStateException with classloader-source context and the ServiceConfigurationError as cause, instead of the raw error propagating with no context (from a static initializer both variants surface as ExceptionInInitializerError). Both paths failed startup/class-init before and still do. Error precedence can also shift: providers are fully enumerated before caller-side validation runs.

Behavior change / release note

• New extension surface (always on, no flag): server startup now scans the application classpath and every plugin classloader for META-INF/services/org.apache.pinot.core.operator.transform.function.TransformFunction descriptors. Deployments without such descriptors are completely unaffected — today no jar in the Pinot distribution ships one.
• New fail-fast startup mode: a malformed descriptor, a provider that cannot be constructed, a null/blank function name, or a canonical-name collision now fails server startup (previously such descriptors were inert because nothing loaded them). This is deliberate: silently skipping a broken provider would surface later as Unsupported function at query time. Operators upgrading with third-party jars that happen to ship such a descriptor should validate them; there is intentionally no disable flag, matching the OperatorTypeDescriptor/RuleSetCustomizer discovery behavior.
• Rolling upgrade: no wire-protocol, serialization, or config-format changes; roll-forward/roll-back safe. Explicit pinot.server.transforms semantics are unchanged.
• The descriptor file name (META-INF/services/org.apache.pinot.core.operator.transform.function.TransformFunction) becomes a permanent external contract once released; a future move of TransformFunction into an SPI module would need a compatibility shim for the old descriptor name.
• getAllFunctions() now returns an immutable snapshot current at call time (previously an unmodifiable live view); no in-repo production callers are affected, and the semantics are documented on the method.
• Docs follow-up: the transform-function docs page should mention the META-INF/services registration path alongside pinot.server.transforms (pinot-docs PR).

Testing

• TransformFunctionFactoryTest (pinot-core, 27 tests): application-classpath discovery, discovery through a real PluginManager plugin realm, factory resolution + ValueBlock evaluation, case/underscore canonicalization, overlapping-classloader dedup, version-skew dedup (child-first classloader fixture) including a skewed copy of a built-in class, identical-class idempotency, discovered-vs-discovered and discovered-vs-built-in collisions, scalar-name shadowing, malformed descriptors, constructor failures, inaccessible constructors, abstract providers, null/blank names, built-ins preserved, legacy pinot.server.transforms registration + explicit overrides, no partial registry publication under concurrent reads, and confirmation that registration never calls init()/evaluation.
• PluginManagerServiceProviderTest (pinot-spi, 8 tests): context-classloader discovery, plugin-realm discovery, cross-classloader dedup, version-skew handling, discovery order, malformed-descriptor and constructor-failure wrapping (source + cause asserted), empty result.
• ServerInstanceTransformFunctionTest (pinot-server, 3 tests): a service-provided transform registers through the (extracted) ServerInstance.initTransformFunctions production path without pinot.server.transforms; the legacy config path and its missing-class failure behavior still work.
• OperatorTypeRegistryTest (7 tests) gains direct unit coverage of the plugin-id validation (registerPlugin is now @VisibleForTesting); PinotRuleSetTest and PluginRealmExportTest pass unchanged on the migrated call sites.
• Service descriptors are generated into temp dirs behind isolated URLClassLoaders (shared ChildFirstClassLoader test utility in the pinot-spi test-jar for the version-skew cases), so no META-INF/services fixture pollutes the test JVM or downstream test-jars.

spotless:check, checkstyle:check, license:check, and git diff --check are clean on all touched modules.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.75258% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.14%. Comparing base (f5fee8e) to head (846e88c).

Files with missing lines	Patch %	Lines
...r/transform/function/TransformFunctionFactory.java	94.00%	2 Missing and 1 partial ⚠️
...rg/apache/pinot/server/starter/ServerInstance.java	25.00%	3 Missing ⚠️
...t/query/runtime/operator/OperatorTypeRegistry.java	77.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19259      +/-   ##
============================================
+ Coverage     67.12%   67.14%   +0.01%     
  Complexity     1424     1424              
============================================
  Files          3462     3462              
  Lines        220677   220743      +66     
  Branches      35255    35266      +11     
============================================
+ Hits         148136   148217      +81     
+ Misses        60708    60706       -2     
+ Partials      11833    11820      -13     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (?)
java-25	67.14% <91.75%> (+0.01%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.14% <91.75%> (+0.01%)	⬆️
unittests	67.14% <91.75%> (+0.01%)	⬆️
unittests1	57.84% <94.62%> (+0.02%)	⬆️
unittests2	39.14% <47.42%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds a reusable PluginManager#loadServiceProviders(Class) helper in pinot-spi and uses it to enable ServiceLoader-based, automatic discovery/registration of external block-oriented TransformFunction implementations during server startup (plus migrating other existing ServiceLoader call sites to the shared helper).

Changes:

• Introduced PluginManager#loadServiceProviders(Class) and PluginManager.ServiceProvider<S> to enumerate providers from the thread context classloader and plugin classloaders with de-dup + improved error context.
• Updated TransformFunctionFactory to build/publish an immutable, atomically swapped registry snapshot that includes built-ins + discovered providers + explicit pinot.server.transforms overrides.
• Migrated PinotRuleSet and OperatorTypeRegistry discovery to the shared helper and expanded unit coverage across affected modules.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pinot-spi/src/main/java/org/apache/pinot/spi/plugin/PluginManager.java	Adds the shared ServiceLoader enumeration helper with de-dup/version-skew handling and source-context errors.
pinot-spi/src/test/java/org/apache/pinot/spi/plugin/PluginManagerServiceProviderTest.java	Unit tests for loadServiceProviders() behavior (order, de-dup, skew, error wrapping).
pinot-spi/src/test/java/org/apache/pinot/spi/plugin/ChildFirstClassLoader.java	Test utility to simulate version-skewed provider classes across classloaders.
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunctionFactory.java	Implements ServiceLoader-based discovery + explicit overrides, publishes immutable registry snapshots, and updates lookup path.
pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/TransformFunctionFactoryTest.java	Comprehensive tests for discovery/override/collision/version-skew + snapshot publication guarantees.
pinot-server/src/main/java/org/apache/pinot/server/starter/ServerInstance.java	Ensures transform functions are initialized as part of server startup via extracted helper method.
pinot-server/src/test/java/org/apache/pinot/server/starter/ServerInstanceTransformFunctionTest.java	Verifies the production server init path registers service-provided transforms without config.
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/rules/PinotRuleSet.java	Migrates RuleSetCustomizer discovery to PluginManager#loadServiceProviders.
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/OperatorTypeRegistry.java	Migrates OperatorTypeDescriptor discovery to the shared helper; factors plugin validation into a testable method.
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/OperatorTypeRegistryTest.java	Adds direct unit coverage for plugin-id validation paths.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.