Skip to content

[Task]: Add a GCS temp bucket cleaner to .test-infra/tools to prevent bucket leaks #40347

Description

@shunping

What needs to happen?

Context

In PR #40279, we started randomizing bucket names for GcsUtil tests to prevent collisions between concurrent test runs. The temporary buckets are now created with randomized names, such as apache-beam-temp-bucket-<UUID>.

Problem

While this randomization successfully resolves naming conflicts, it introduces a risk of bucket leaks. If a test crashes, is interrupted, or fails to clean up properly after execution, these randomized buckets will be left behind in the project indefinitely, unnecessarily consuming resources.

Possible Solution

As discussed in this review comment, we should build a mechanism to automatically sweep up these orphaned buckets:

  • Create a cleanup tool: Add a GCS temp bucket cleaner script or utility in the /.test-infra/tools directory. The tool should be responsible for identifying and deleting stale apache-beam-temp-bucket-* buckets (e.g., buckets older than a certain number of hours).
  • Integrate with CI: Add this cleaner as a step in the existing cleanUp workflow so that it runs regularly to automatically eliminate any leaked buckets.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions