Skip to content

[filesystem][s3] Serve s3a:// paths with the S3 plugin - #9814

Open
dev-donghwan wants to merge 1 commit into
apache:masterfrom
dev-donghwan:paimon-s3a-scheme
Open

[filesystem][s3] Serve s3a:// paths with the S3 plugin#9814
dev-donghwan wants to merge 1 commit into
apache:masterfrom
dev-donghwan:paimon-s3a-scheme

Conversation

@dev-donghwan

Copy link
Copy Markdown

Purpose

paimon-s3 registers only the s3 scheme, so a table whose location starts with s3a:// cannot be opened by the plugin:

discovered schemes = [s3, viewfs, file, hdfs]
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation
for scheme 's3a' in the classpath. Hadoop FileSystem also cannot access this path
's3a://<bucket>/t/x.txt'.

s3a:// is not an exotic spelling, it is what a Hadoop 3 deployment produces. The S3 and S3N clients were removed from Hadoop 3, and the Hadoop copy bundled inside this very plugin shows it: core-default.xml declares fs.s3a.impl only, and the one remaining s3n class fails with The s3n:// client to Amazon S3 is no longer available: please migrate to the s3a:// client. A Hive Metastore therefore stores s3a:// locations, and every engine that resolves a table through HMS hands that path to Paimon.

Engines that ship their own FileIO can paper over this with a fallback loader, but the ones that do not, Spark and StarRocks for example, have nothing to fall back to (see #7835 for the same failure on OBS). The documented alternative, location-in-properties, moves the location out of the Hive StorageDescriptor into table properties, which keeps the path away from Hive's own filesystem but also hides it from every reader that looks at the standard field, and it has its own history of problems (#2729, #3309).

The fix is small because both schemes already share one implementation:

  • S3FileIO.createFileSystem builds new S3AFileSystem() and initializes it with the URI as given, so nothing in the plugin depends on the scheme spelling.
  • S3AFileSystem.getScheme() returns a field that is set from that URI, not a constant.
  • S3FileIO.CONFIG_PREFIXES already accepts s3., s3a. and fs.s3a., and maps all of them onto fs.s3a.*.

So this PR adds an S3ALoader that reports s3a and reuses S3Loader unchanged. FileIO.discoverLoaders keys loaders by scheme and only rejects duplicates of the same scheme, so registering a second loader needs no interface change.

For what it is worth, PyPaimon already treats the schemes as equivalent (pyarrow_file_io.py: elif scheme in {"s3", "s3a", "s3n"}), so this also brings the Java side in line.

Tests

S3ASchemeTest, backed by the existing MinIO container, five cases:

  • s3 and s3a are both present in the loader table and map to their own loaders
  • write, read, list and delete over s3a://
  • an object written through s3:// is readable through s3a://
  • the Hadoop filesystem actually created is S3AFileSystem for both schemes, asserted by unwrapping the plugin FileIO
[s3]  hadoop FileSystem = org.apache.hadoop.fs.s3a.S3AFileSystem, getScheme() = s3,  getUri() = s3://bucket-jhrebf
[s3a] hadoop FileSystem = org.apache.hadoop.fs.s3a.S3AFileSystem, getScheme() = s3a, getUri() = s3a://bucket-jhrebf
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

The test module needed hadoop-hdfs-client and paimon-hadoop-shaded-3.4 in test scope, because CatalogContext loads HdfsConfiguration.

API and Backward Compatibility

No interface change and no behaviour change for s3:// paths. Paths that used to fail with UnsupportedSchemeException now resolve to the same FileIO the s3 scheme has always used.

Documentation

The supported filesystems table now lists s3://, s3a:// for S3.

paimon-s3 registers only the s3 scheme, so a table whose location uses
s3a:// fails with UnsupportedSchemeException unless the engine provides
a fallback FileIO. Hive Metastore and every Hadoop 3 client write s3a://
locations, because the s3 and s3n clients were removed from Hadoop 3.

The plugin already talks to S3 through S3AFileSystem and maps its s3.*
options onto fs.s3a.*, and S3AFileSystem keeps the scheme of the URI it
was initialized with, so the two schemes share one code path. Register
an S3ALoader for s3a:// and cover both schemes with tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant