[feat](vault) Support keyless GCS vaults with GKE Workload Identity#65433
Open
cyriltovena wants to merge 7 commits into
Open
[feat](vault) Support keyless GCS vaults with GKE Workload Identity#65433cyriltovena wants to merge 7 commits into
cyriltovena wants to merge 7 commits into
Conversation
…ult Credentials
Storage vaults on GCS currently require static HMAC keys: the only
credential fields an S3-type vault accepts for provider GCP are
s3.access_key/s3.secret_key. Every other credential mode
(role_arn, s3.credentials_provider_type) is AWS-only, so clusters
running on GKE cannot use Workload Identity and must mint and
distribute long-lived HMAC key pairs.
This adds a new credentials provider type, GCP_ADC, valid only for
provider GCP vaults:
CREATE STORAGE VAULT gcs_vault PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);
The GCS XML API accepts OAuth2 bearer tokens as an alternative to
HMAC signatures, so no new SDK dependency is needed:
- A shared GcpAdcTokenProvider (common/cpp) resolves tokens following
the Application Default Credentials order: a service account json
pointed to by GOOGLE_APPLICATION_CREDENTIALS (JWT assertion signed
with OpenSSL, exchanged at the OAuth2 token endpoint), else the
GCE/GKE metadata server (i.e. Workload Identity). Tokens are cached
and refreshed ahead of expiry. Built on the AWS SDK's own HTTP/JSON
utilities.
- BE S3ObjStorageClient and recycler S3ObjClient attach the token as
an Authorization: Bearer header per request; the S3 client is
created with anonymous credentials so SigV4 signing is skipped
(mirroring how the existing keyless credential providers reuse the
provider seam added for vaults without role ARN).
- FE validates the property combination (provider GCP only, mutually
exclusive with ak/sk and role_arn) and skips its static-credential
connectivity probe; the BE probes the vault at startup instead,
like assumed-role vaults.
- Meta-service accepts keyless GCP vaults on ADD_S3_VAULT and
ALTER_S3_VAULT (switching an existing HMAC vault to GCP_ADC clears
the stored keys, symmetric with the ak/sk <-> role_arn switches).
Presigned URLs are not supported in this mode (a bearer token cannot
be embedded in a URL) and return an empty string with a warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cyriltovena
requested review from
Gabriel39,
gavinchou,
liaoxin01,
luwei16 and
morningman
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
/review |
Author
|
run buildall |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29324 ms |
Author
|
run feut |
Contributor
TPC-DS: Total hot run time: 180486 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
ClickBench: Total hot run time: 25.61 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29454 ms |
Contributor
TPC-DS: Total hot run time: 180054 ms |
Contributor
ClickBench: Total hot run time: 25.02 s |
Author
|
run p0 |
cyriltovena
changed the title
Author
|
run buildall |
Author
|
run feut |
Author
|
run p0 |
Author
|
/review |
1 similar comment
Author
|
/review |
Author
|
run p0 |
Author
|
run feut |
Author
|
/review |
Author
|
run buildall |
Author
|
/review |
Author
|
/review |
Author
|
run feut |
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29557 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-DS: Total hot run time: 179704 ms |
Contributor
ClickBench: Total hot run time: 25.85 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65432
Related PRs: #62788, #64766
Problem:
GCP storage vaults currently require long-lived HMAC access keys. GKE deployments using Workload Identity Federation have no supported way to authenticate the vault data path with the pod's short-lived Google identity.
Solution:
Add
gcp_workload_identity, a GCP-only storage-vault credentials provider. BE and recycler obtain short-lived OAuth2 tokens from the GKE/GCE metadata server and attach them to GCS XML API requests while leaving AWS SigV4 signing disabled.CREATE STORAGE VAULT gcs_vault PROPERTIES ( "type" = "S3", "s3.endpoint" = "storage.googleapis.com", "s3.region" = "us-central1", "s3.bucket" = "my-bucket", "s3.root.path" = "doris", "provider" = "GCP", "s3.credentials_provider_type" = "gcp_workload_identity" );The implementation:
CredProviderTypePB::GCP_WORKLOAD_IDENTITYfor storage-vault protobuf propagation. The legacy S3 resource/Thrift path remains unchanged.NotSupportedresult when a caller requests a presigned URL, rather than returning an empty string.Security Impact:
This removes the need to distribute long-lived HMAC keys on GKE. Workload Identity is restricted to
provider = GCPand the canonicalhttps://storage.googleapis.comendpoint at FE, meta-service, BE, and recycler boundaries. HTTP and arbitrary endpoints are rejected so OAuth bearer tokens cannot be sent to an untrusted destination. Token acquisition failure also fails closed instead of silently falling back to anonymous access.Testing:
git diff --check, clang-format validation, and protobuf syntax validation pass.Release note
Support keyless GCS storage vault authentication on GKE with Workload Identity Federation using
"s3.credentials_provider_type" = "gcp_workload_identity".Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)