Skip to content

fix(storage): allow setting full checksum at finalize for appendable uploads#16251

Open
v-pratap wants to merge 2 commits into
googleapis:mainfrom
v-pratap:append-full-object-checksum
Open

fix(storage): allow setting full checksum at finalize for appendable uploads#16251
v-pratap wants to merge 2 commits into
googleapis:mainfrom
v-pratap:append-full-object-checksum

Conversation

@v-pratap

Copy link
Copy Markdown
Contributor

Fixes the checksum validation logic at the end of appendable uploads so users can provide a final expected CRC.

@v-pratap v-pratap requested review from a team as code owners July 11, 2026 07:22
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Jul 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the AsyncWriterConnectionImpl::Finalize method to handle manual CRC32C checksum injection via UseCrc32cValueOption and to avoid sending partial CRCs for appendable uploads. The reviewer identified a critical inconsistency in how current_options and options_ are checked for the CRC option, which could lead to the option being ignored or the upload failing. A consistent check across both option sources was suggested to resolve this issue.

Comment on lines +145 to +161
auto current_options = google::cloud::internal::CurrentOptions();

// Default to letting the internal hash function compute and send the checksum.
auto action = PartialUpload::kFinalizeWithChecksum;

if (current_options.has<google::cloud::storage::UseCrc32cValueOption>()) {
// The user provided a final CRC via OptionsSpan. We manually inject it into the
// request. We use `kFinalize` so the internal hash function doesn't overwrite it.
write.mutable_object_checksums()->set_crc32c(
current_options.get<google::cloud::storage::UseCrc32cValueOption>());
action = PartialUpload::kFinalize;
} else if (is_append && !options_->has<google::cloud::storage::UseCrc32cValueOption>()) {
// For appendable uploads, the internal hash function only sees the chunks uploaded
// in this stream, not the full object. We use `kFinalize` to avoid sending this
// partial CRC, which would otherwise fail validation.
action = PartialUpload::kFinalize;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is an inconsistency in how current_options and options_ are checked. If UseCrc32cValueOption is provided via the connection's options_ but not in the thread-local current_options, the first if condition is skipped, and the else if condition is also skipped (since options_->has<...>() is true). This results in action remaining kFinalizeWithChecksum without actually setting the CRC on the request, which will cause the upload to fail or ignore the user-provided CRC.

We should check both current_options and options_ consistently for the CRC option.

  auto const& current_options = google::cloud::internal::CurrentOptions();
  auto has_crc_option = current_options.has<google::cloud::storage::UseCrc32cValueOption>() ||
                        options_->has<google::cloud::storage::UseCrc32cValueOption>();

  // Default to letting the internal hash function compute and send the checksum.
  auto action = PartialUpload::kFinalizeWithChecksum;

  if (has_crc_option) {
    // The user provided a final CRC. We manually inject it into the request.
    // We use kFinalize so the internal hash function doesn't overwrite it.
    auto crc = current_options.has<google::cloud::storage::UseCrc32cValueOption>()
                   ? current_options.get<google::cloud::storage::UseCrc32cValueOption>()
                   : options_->get<google::cloud::storage::UseCrc32cValueOption>();
    write.mutable_object_checksums()->set_crc32c(crc);
    action = PartialUpload::kFinalize;
  } else if (is_append) {
    // For appendable uploads, the internal hash function only sees the chunks uploaded
    // in this stream, not the full object. We use kFinalize to avoid sending this
    // partial CRC, which would otherwise fail validation.
    action = PartialUpload::kFinalize;
  }

@v-pratap v-pratap force-pushed the append-full-object-checksum branch from 7e5ced2 to 982d3bf Compare July 11, 2026 07:24
@v-pratap v-pratap force-pushed the append-full-object-checksum branch from aab03ae to bf842c3 Compare July 11, 2026 08:21
@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.05825% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.26%. Comparing base (4d78d42) to head (bf842c3).

Files with missing lines Patch % Lines
...rage/internal/async/writer_connection_impl_test.cc 97.84% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #16251   +/-   ##
=======================================
  Coverage   92.26%   92.26%           
=======================================
  Files        2214     2214           
  Lines      206307   206406   +99     
=======================================
+ Hits       190348   190447   +99     
  Misses      15959    15959           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant