[filesystem][s3] Serve s3a:// paths with the S3 plugin - #9814
Open
dev-donghwan wants to merge 1 commit into
Open
[filesystem][s3] Serve s3a:// paths with the S3 plugin#9814dev-donghwan wants to merge 1 commit into
dev-donghwan wants to merge 1 commit into
Conversation
paimon-s3 registers only the s3 scheme, so a table whose location uses s3a:// fails with UnsupportedSchemeException unless the engine provides a fallback FileIO. Hive Metastore and every Hadoop 3 client write s3a:// locations, because the s3 and s3n clients were removed from Hadoop 3. The plugin already talks to S3 through S3AFileSystem and maps its s3.* options onto fs.s3a.*, and S3AFileSystem keeps the scheme of the URI it was initialized with, so the two schemes share one code path. Register an S3ALoader for s3a:// and cover both schemes with tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
paimon-s3registers only thes3scheme, so a table whose location starts withs3a://cannot be opened by the plugin:s3a://is not an exotic spelling, it is what a Hadoop 3 deployment produces. The S3 and S3N clients were removed from Hadoop 3, and the Hadoop copy bundled inside this very plugin shows it:core-default.xmldeclaresfs.s3a.implonly, and the one remainings3nclass fails withThe s3n:// client to Amazon S3 is no longer available: please migrate to the s3a:// client. A Hive Metastore therefore storess3a://locations, and every engine that resolves a table through HMS hands that path to Paimon.Engines that ship their own
FileIOcan paper over this with a fallback loader, but the ones that do not, Spark and StarRocks for example, have nothing to fall back to (see #7835 for the same failure on OBS). The documented alternative,location-in-properties, moves the location out of the HiveStorageDescriptorinto table properties, which keeps the path away from Hive's own filesystem but also hides it from every reader that looks at the standard field, and it has its own history of problems (#2729, #3309).The fix is small because both schemes already share one implementation:
S3FileIO.createFileSystembuildsnew S3AFileSystem()and initializes it with the URI as given, so nothing in the plugin depends on the scheme spelling.S3AFileSystem.getScheme()returns a field that is set from that URI, not a constant.S3FileIO.CONFIG_PREFIXESalready acceptss3.,s3a.andfs.s3a., and maps all of them ontofs.s3a.*.So this PR adds an
S3ALoaderthat reportss3aand reusesS3Loaderunchanged.FileIO.discoverLoaderskeys loaders by scheme and only rejects duplicates of the same scheme, so registering a second loader needs no interface change.For what it is worth, PyPaimon already treats the schemes as equivalent (
pyarrow_file_io.py:elif scheme in {"s3", "s3a", "s3n"}), so this also brings the Java side in line.Tests
S3ASchemeTest, backed by the existing MinIO container, five cases:s3ands3aare both present in the loader table and map to their own loaderss3a://s3://is readable throughs3a://S3AFileSystemfor both schemes, asserted by unwrapping the pluginFileIOThe test module needed
hadoop-hdfs-clientandpaimon-hadoop-shaded-3.4in test scope, becauseCatalogContextloadsHdfsConfiguration.API and Backward Compatibility
No interface change and no behaviour change for
s3://paths. Paths that used to fail withUnsupportedSchemeExceptionnow resolve to the sameFileIOthes3scheme has always used.Documentation
The supported filesystems table now lists
s3://, s3a://for S3.