fix(storage): allow setting full checksum at finalize for appendable uploads#16251
fix(storage): allow setting full checksum at finalize for appendable uploads#16251v-pratap wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the AsyncWriterConnectionImpl::Finalize method to handle manual CRC32C checksum injection via UseCrc32cValueOption and to avoid sending partial CRCs for appendable uploads. The reviewer identified a critical inconsistency in how current_options and options_ are checked for the CRC option, which could lead to the option being ignored or the upload failing. A consistent check across both option sources was suggested to resolve this issue.
| auto current_options = google::cloud::internal::CurrentOptions(); | ||
|
|
||
| // Default to letting the internal hash function compute and send the checksum. | ||
| auto action = PartialUpload::kFinalizeWithChecksum; | ||
|
|
||
| if (current_options.has<google::cloud::storage::UseCrc32cValueOption>()) { | ||
| // The user provided a final CRC via OptionsSpan. We manually inject it into the | ||
| // request. We use `kFinalize` so the internal hash function doesn't overwrite it. | ||
| write.mutable_object_checksums()->set_crc32c( | ||
| current_options.get<google::cloud::storage::UseCrc32cValueOption>()); | ||
| action = PartialUpload::kFinalize; | ||
| } else if (is_append && !options_->has<google::cloud::storage::UseCrc32cValueOption>()) { | ||
| // For appendable uploads, the internal hash function only sees the chunks uploaded | ||
| // in this stream, not the full object. We use `kFinalize` to avoid sending this | ||
| // partial CRC, which would otherwise fail validation. | ||
| action = PartialUpload::kFinalize; | ||
| } |
There was a problem hiding this comment.
There is an inconsistency in how current_options and options_ are checked. If UseCrc32cValueOption is provided via the connection's options_ but not in the thread-local current_options, the first if condition is skipped, and the else if condition is also skipped (since options_->has<...>() is true). This results in action remaining kFinalizeWithChecksum without actually setting the CRC on the request, which will cause the upload to fail or ignore the user-provided CRC.
We should check both current_options and options_ consistently for the CRC option.
auto const& current_options = google::cloud::internal::CurrentOptions();
auto has_crc_option = current_options.has<google::cloud::storage::UseCrc32cValueOption>() ||
options_->has<google::cloud::storage::UseCrc32cValueOption>();
// Default to letting the internal hash function compute and send the checksum.
auto action = PartialUpload::kFinalizeWithChecksum;
if (has_crc_option) {
// The user provided a final CRC. We manually inject it into the request.
// We use kFinalize so the internal hash function doesn't overwrite it.
auto crc = current_options.has<google::cloud::storage::UseCrc32cValueOption>()
? current_options.get<google::cloud::storage::UseCrc32cValueOption>()
: options_->get<google::cloud::storage::UseCrc32cValueOption>();
write.mutable_object_checksums()->set_crc32c(crc);
action = PartialUpload::kFinalize;
} else if (is_append) {
// For appendable uploads, the internal hash function only sees the chunks uploaded
// in this stream, not the full object. We use kFinalize to avoid sending this
// partial CRC, which would otherwise fail validation.
action = PartialUpload::kFinalize;
}7e5ced2 to
982d3bf
Compare
aab03ae to
bf842c3
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16251 +/- ##
=======================================
Coverage 92.26% 92.26%
=======================================
Files 2214 2214
Lines 206307 206406 +99
=======================================
+ Hits 190348 190447 +99
Misses 15959 15959 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Fixes the checksum validation logic at the end of appendable uploads so users can provide a final expected CRC.