Skip to content

feat: handle parallel chunk upload assembly race in Local and S3 devices#163

Merged
TorstenDittmann merged 1 commit intomainfrom
feat-parallel-chunk-uploads
Apr 29, 2026
Merged

feat: handle parallel chunk upload assembly race in Local and S3 devices#163
TorstenDittmann merged 1 commit intomainfrom
feat-parallel-chunk-uploads

Conversation

@TorstenDittmann
Copy link
Copy Markdown
Contributor

Summary

This PR adds race-condition handling for parallel chunk uploads across all storage devices.

Problem

When two requests upload the final missing chunks in parallel, both can observe that all chunks are present and attempt to assemble/complete the file simultaneously. The loser would previously throw an error even though the file is fully assembled.

Local Device

  • Added file_exists($path) guard at the start of joinChunks() — if the final file already exists, another request already assembled it, so return silently.
  • Added file_exists($path) guard in the rename() failure handler — if rename() fails because the destination already exists, clean up the temp assembly file and return silently instead of throwing.

S3 Device (and all S3-compatible adapters: AWS, Backblaze, DO Spaces, Linode, Wasabi)

  • Added $this->exists($path) guard before completeMultipartUpload() — if the final object already exists, another request already completed the multipart upload, so return the chunk count silently.

Tests

  • Added testParallelChunkUpload() to LocalTest.php to verify that calling joinChunks() when the file already exists does not throw and does not corrupt the file.

Backwards Compatibility

Fully backwards compatible. The changes only affect error paths when assembly/completion races occur. No public API signatures are changed.

Related

  • Prerequisite for the Appwrite server PR that enables parallel chunk uploads at the API layer.

@TorstenDittmann TorstenDittmann force-pushed the feat-parallel-chunk-uploads branch 2 times, most recently from bedcdc3 to b9a672f Compare April 28, 2026 10:40
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Apr 28, 2026

Greptile Summary

This PR adds race-condition guards for parallel chunk upload assembly in the Local and S3 devices: a file_exists early-return at the top of joinChunks(), a tempnam()-based unique assembly path, a rename()-failure fallback guard, and an exists() check before completeMultipartUpload in S3. The two new P2 findings are a missing false-check on tempnam() and a misleading test method name; the more significant issues (S3 header state mutation on early return, orphaned multipart upload, and TOCTOU on the Local POSIX rename path) were raised in prior review rounds and remain unresolved.

Confidence Score: 3/5

Mergeable for the Local device path; the S3 path still has unresolved P1 issues from prior review rounds that should be addressed before shipping.

No new P1s found in this pass, but pre-existing P1s from earlier rounds — S3 headers left mutated on the early-return path, and the active $uploadId multipart upload abandoned without abort — remain unresolved in the current diff. Multiple unresolved P1s pull the score below the 4/5 ceiling.

src/Storage/Device/S3.php — header restore is unreachable on the early-return path and the multipart upload is left orphaned.

Important Files Changed

Filename Overview
src/Storage/Device/Local.php Adds file_exists early-return guard and switches to tempnam() for unique assembly paths; tempnam() return value is not checked for false.
src/Storage/Device/S3.php Adds exists() guard before completeMultipartUpload with header save/restore; headers are NOT restored on the early-return path and the current $uploadId multipart upload is left orphaned (both issues raised in prior review threads).
tests/Storage/Device/LocalTest.php Adds testParallelChunkUpload() covering the top-level file_exists guard; existing test method name now misrepresents the tempnam() bypass-not-overwrite behaviour.
.gitignore Adds .DS_Store to ignore list and fixes missing trailing newline — trivial housekeeping.

Reviews (6): Last reviewed commit: "feat: handle parallel chunk upload assem..." | Re-trigger Greptile

Comment thread src/Storage/Device/Local.php
Comment thread src/Storage/Device/S3.php
@TorstenDittmann TorstenDittmann force-pushed the feat-parallel-chunk-uploads branch from b9a672f to 8c524af Compare April 28, 2026 11:02
Comment thread src/Storage/Device/S3.php
@TorstenDittmann TorstenDittmann force-pushed the feat-parallel-chunk-uploads branch 2 times, most recently from b97e42e to fb3ec61 Compare April 28, 2026 13:38
Comment thread src/Storage/Device/S3.php
@TorstenDittmann TorstenDittmann force-pushed the feat-parallel-chunk-uploads branch 2 times, most recently from f1ee72f to 184f505 Compare April 28, 2026 14:32
@TorstenDittmann TorstenDittmann force-pushed the feat-parallel-chunk-uploads branch from 184f505 to 65b3b3e Compare April 28, 2026 14:55
@TorstenDittmann TorstenDittmann merged commit 8a2e3a8 into main Apr 29, 2026
8 of 9 checks passed
@TorstenDittmann TorstenDittmann deleted the feat-parallel-chunk-uploads branch April 29, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants