Skip to content

[filesystem] Isolate Jindo OSS presigning dependencies - #9856

Open
JunRuiLee wants to merge 1 commit into
apache:masterfrom
JunRuiLee:fix/jindo-oss-plugin-isolation-master
Open

JunRuiLee wants to merge 1 commit into
apache:masterfrom
JunRuiLee:fix/jindo-oss-plugin-isolation-master

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Purpose

Prevent the OSS SDK dependency closure bundled in paimon-jindo from shadowing libraries supplied by an embedding application. For example, the bundled Gson 2.8.6 can mask a newer Gson used by Doris, depending on classpath order.

Reuse the same PluginLoader and private-directory classloader isolation mechanism already used by paimon-oss, rather than maintaining a package-relocation list. This applies to the OSS presigning implementation and its dependency closure; it does not change the Jindo/Hadoop filesystem loading path or introduce a separate implementation Maven module.

Specifically:

  • Load the OSS presigning implementation, SDK dependencies, helper, and resources from paimon-plugin-jindo-oss/ instead of exposing them at the JAR root.
  • Keep the Jindo/Hadoop boundary unchanged. Share Paimon types and logging APIs with the host, and retain the existing Java 11 JAXB multi-release entries.
  • Initialize the presigner lazily and restore the thread context classloader after initialization, signing, and shutdown.
  • Add packaged-artifact regression tests without introducing another Maven module or modifying the common classloader implementation.

This PR targets master; a separate release-2.1 backport can follow after the mainline change is accepted.

Tests

Ran the following on JDK 8 and JDK 11 against this master-based commit:

mvn -B -ntp -pl paimon-filesystems/paimon-jindo -am \
  -Dmaven.jar.forceCreation=true \
  '-Dtest=JindoFileIOTest#testPlugin*+testCreateBlob*+testDisable*+testKeep*' \
  -DwildcardSuites=none -DfailIfNoTests=false verify

Both runs pass: 4 selected unit tests and 4 packaged-artifact isolation tests per JDK, with Checkstyle, Spotless, and Enforcer enabled.

The artifact tests cover root/private JAR contents and SPI discovery, a minimal host classpath, conflicting dependencies before and after the plugin JAR, independent Gson serialization, private OSS resources, shared logging classes, classloader-safe reflection, presigning, and context-classloader restoration.

The module Apache RAT check also passes (15 approved files, 0 unapproved), excluding only the generated dependency-reduced-pom.xml. git diff --check passes.

Local validation does not include a live Doris/OSS deployment. The existing native-filesystem fallback test was not selected because the local macOS/arm64 environment cannot run the Jindo native filesystem; the test itself remains enabled for supported environments.

@JunRuiLee
JunRuiLee force-pushed the fix/jindo-oss-plugin-isolation-master branch from 5bbec4e to ca5fe46 Compare September 15, 2026 16:30
@JunRuiLee
JunRuiLee marked this pull request as draft September 15, 2026 17:56
@JunRuiLee
JunRuiLee marked this pull request as ready for review September 16, 2026 01:56
@JunRuiLee
JunRuiLee marked this pull request as draft September 16, 2026 02:05
Load OSS presigning dependencies from a private plugin directory with the existing PluginLoader mechanism, without adding a Maven module or changing the shared classloader implementation.

Keep commons-logging available privately, initialize the plugin loader lazily, and preserve Jindo-specific dependency notices and JDOM/JAXB license resources.

Use the existing unit/integration test naming convention to verify packaged class and resource isolation, minimal and conflicting host classpaths, multi-release entries, and presigner class boundaries.
@JunRuiLee
JunRuiLee force-pushed the fix/jindo-oss-plugin-isolation-master branch from 578feef to 9ed877f Compare September 16, 2026 03:09
@JunRuiLee
JunRuiLee marked this pull request as ready for review September 16, 2026 03:16
@JunRuiLee

Copy link
Copy Markdown
Contributor Author

This was found while integrating Paimon 2.1 into Doris: the OSS SDK bundled at the root of
paimon-jindo shadows libraries the host provides — the bundled Gson 2.8.6 masks the newer
Gson used by Doris, depending on classpath order.

The fix moves it into a private plugin directory via the existing PluginLoader, same as
paimon-oss. No new Maven module, no change to the shared classloader.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant