feat: add OpenDAL-backed GooseFS object store support - #9134
Open
XuQianJin-Stars wants to merge 2 commits into
Open
feat: add OpenDAL-backed GooseFS object store support#9134XuQianJin-Stars wants to merge 2 commits into
XuQianJin-Stars wants to merge 2 commits into
Conversation
Add `goosefs://` scheme support to `vortex-cloud`, bridging Tencent Cloud GooseFS through OpenDAL's `services::GooseFs` via `object_store_opendal`. Rust changes: - `vortex-cloud`: new `goosefs` Cargo feature and `opendal::goosefs` module with `GoosefsConfig`, `make_goosefs_store`, and URL-to-config translation (master_addr from properties -> URL authority -> GOOSEFS_MASTER_ADDR env) - `vortex-cloud/src/registry`: extend OpenDAL cfg gates to include `goosefs` - 7 unit tests covering missing master_addr, URL authority, env fallback, property override, HA mode, and explicit config construction Python bindings: - `vortex-python`: new `GoosefsStore` pyclass with keyword-only options (root, block_size, chunk_size, write_type, auth_type, auth_username) - `AnyVortexStore` accepts `GoosefsStore` in `read_url` / `write` - Python store module `vortex.store._goosefs` with graceful ImportError fallback when the `opendal` feature is not enabled - Type stub (`.pyi`) for `GoosefsStore` constructor Dependency note: goosefs-sdk v0.1.8 is pulled transitively via opendal-service-goosefs v0.57.0; no direct workspace dependency added. Signed-off-by: forwardxu <forwardxu@apache.org>
XuQianJin-Stars
force-pushed
the
feat/opendal-goosefs
branch
from
August 1, 2026 04:04
c6097cf to
a6071f6
Compare
Contributor
Author
|
Hi @robert3005, could you please help review this PR when you have a moment? Thanks! |
The test_opendal_schemes_reach_the_opendal_builder test assumed all OpenDAL-backed schemes fail to build without an endpoint. GooseFS is different: its master_addr is derived from the URL authority, so goosefs://bucket/key.vortex builds successfully. Update the test to accept Ok (proof the OpenDAL builder ran, since parse_url_opts rejects goosefs://) in addition to an OpenDAL error, and add goosefs to the cfg gate. Signed-off-by: forwardxu <forwardxu@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
goosefs://scheme support tovortex-cloud, bridging Tencent Cloud GooseFS through OpenDAL'sservices::GooseFsviaobject_store_opendal.Rust changes:
vortex-cloud: newgoosefsCargo feature andopendal::goosefsmodule withGoosefsConfig,make_goosefs_store, and URL-to-config translation (master_addr from properties -> URL authority -> GOOSEFS_MASTER_ADDR env)vortex-cloud/src/registry: extend OpenDAL cfg gates to includegoosefsPython bindings:
vortex-python: newGoosefsStorepyclass with keyword-only options (root, block_size, chunk_size, write_type, auth_type, auth_username)AnyVortexStoreacceptsGoosefsStoreinread_url/writevortex.store._goosefswith graceful ImportError fallback when theopendalfeature is not enabled.pyi) forGoosefsStoreconstructorDependency note: goosefs-sdk v0.1.8 is pulled transitively via opendal-service-goosefs v0.57.0; no direct workspace dependency added.
Rationale for this change
What changes are included in this PR?
This PR introduces a new OpenDAL-backed object store scheme in
vortex-cloud:goosefs(gated under the existingopendalumbrella).vortex-cloud/src/opendal/goosefs.rs(~300 lines) containing:GoosefsConfigstruct (master_addr, root, block_size, chunk_size, write_type, auth_type, auth_username).make_goosefs_store(config)— strongly-typed entry point returningArc<dyn ObjectStore>.url_and_properties_to_config(url, properties, env_lookup)— translates agoosefs://URL plus a property map into aGoosefsConfig, withmaster_addrresolved asproperties["master_addr"]→ URL authority (host:port) →GOOSEFS_MASTER_ADDRenv var.vortex-cloud/src/opendal/mod.rs:mod goosefs;,pub use ... GOOSEFS_SCHEME / GoosefsConfig / make_goosefs_store.SUPPORTED_SCHEMESandmake_opendal_store_with_envmatch arm extended to dispatch ongoosefs://.property_or_env,warn_on_unknown_properties,build_operatorcfg gates updated.supports_scheme_tracks_enabled_featuresandevery_supported_scheme_dispatchestests extended to cover the new scheme.vortex-cloud/src/registry/mod.rs:EnvSource::lookupandRegistry::build_storecfg gates updated so the registry can resolvegoosefs://URLs.vortex-cloud/src/lib.rsdoc comment +opendalfeature list updated.vortex-python:GoosefsStorepyclass invortex-python/src/opendal_store.rswith the same keyword-only options asGoosefsConfig.AnyVortexStoreenum extended with aGoosefs(GoosefsStore)variant, with matchinginto_innerandFromPyObjectarms.vortex-python/python/vortex/store/_goosefs.pymirroring_cos.py(gracefulImportErrorplaceholder when theopendalfeature is disabled).vortex-python/python/vortex/store/__init__.pyupdated: importGoosefsStore, add toObjectStoretype alias, add to__all__, documentgoosefs://URL handling in thefrom_urldocstring.vortex-python/python/vortex/_lib/__init__.pyiupdated with theGoosefsStoreclass stub.Dependency notes:
goosefs-sdkis not added as a direct workspace dependency. It is pulled transitively throughopendal-service-goosefs v0.57.0, which currently usesgoosefs-sdk v0.1.8. Adding a direct dependency would force an extra upgrade every time the OpenDAL facade bumps.developwithopendal = "0.57.0"andobject_store = "0.13.2"(compatible withdatafusion 54).opendalto0.58.1; it depends ondatafusionmoving toobject_store 0.14and is intentionally not folded into this PR.What APIs are changed? Are there any user-facing changes?
Yes — this PR adds new public APIs and a new URL scheme. It does not modify or break any existing public API.
New public Rust API (gated on the new
goosefsfeature)In
vortex-cloud::opendal:pub const GOOSEFS_SCHEME: &str = "goosefs";pub struct GoosefsConfig { pub master_addr: String, pub root: Option<String>, pub block_size: Option<u64>, pub chunk_size: Option<u64>, pub write_type: Option<String>, pub auth_type: Option<String>, pub auth_username: Option<String> }pub fn make_goosefs_store(config: GoosefsConfig) -> Result<Arc<dyn object_store::ObjectStore>, OpenDALStoreError>The existing
make_opendal_store_with_envURL dispatcher now also accepts thegoosefsscheme, returning a store rooted at the URL path.supports_scheme("goosefs")returnstruewhen thegoosefsfeature is enabled (and the corresponding Cargo features are turned on via theopendalumbrella).New public Cargo features
vortex-cloud/goosefs— opt-in. When enabled,opendal,object_store_opendal,tracing, andopendal/services-goosefsare pulled in.vortex-cloud/opendalnow also enables the newgoosefsfeature (alongside the existingcosandoss).New Python API (gated on the
opendalfeature)In
vortex._lib:class GoosefsStore(master_addr: str, *, root: str | None = None, block_size: int | None = None, chunk_size: int | None = None, write_type: str | None = None, auth_type: str | None = None, auth_username: str | None = None)— apyo3-backed store class.In
vortex.store:GoosefsStorere-exported with the same constructor signature.vortex.store.ObjectStoreunion type extended to includeGoosefsStore.vortex.store.from_url(...)recognisesgoosefs://master-addr:port/pathURLs (when theopendalfeature is enabled).vortex.io.read_url(url, store=...)andvortex.io.write(..., store=...)accept aGoosefsStoreinstance.New environment variable
GOOSEFS_MASTER_ADDR— read byGoosefsConfig::from_properties_auto()(the OpenDAL builder) and byvortex-cloud'surl_and_properties_to_configas the final fallback formaster_addrwhen the URL has no authority and the property map is empty.Backwards compatibility
cos,oss,registry, etc.) are unaffected.goosefsfeature is opt-in; existing consumers that don't enable it will see no new code or dependencies pulled in.goosefsscheme is recognised byvortex-cloud/src/registry::Registryonly when thegoosefsfeature is enabled, so existing deployments won't accidentally treatgoosefs://URLs as a different scheme.