Skip to content

[common] Fix snapshot and tag reads failing when the file is deleted concurrently - #9851

Open
sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:fix/snapshot-missing-on-read
Open

sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:fix/snapshot-missing-on-read

Conversation

@sundapeng

@sundapeng sundapeng commented Sep 15, 2026

Copy link
Copy Markdown
Member

Purpose

Paimon treats a snapshot, tag, changelog or schema file as missing only when reading it throws FileNotFoundException. Some FileIOs, such as OSSFileIO and Jindo, throw a plain IOException instead when the file is deleted while it is being read. A writer traversing snapshots in latestSnapshotOfUser then fails its checkpoint when it reaches a snapshot being expired, and TagManager#get fails instead of returning empty for a tag being deleted.

When reading fails with another IOException, FileIO#readFileUtf8 now throws FileNotFoundException if the file is gone, and keeps the original error otherwise.

Tests

FileIOTest: 2 new cases, 7 tests passed. SnapshotManagerTest: 1 new case, 43 tests passed.

@sundapeng
sundapeng force-pushed the fix/snapshot-missing-on-read branch from 7d6bcf7 to b71cbd9 Compare September 15, 2026 11:41
Some FileIOs report a snapshot that is deleted after its input stream was
opened as a generic IOException on the first read, for example
"404 Not Found / NoSuchKey" from an object store. SnapshotManager#tryFromPath
turned that into a RuntimeException, so callers that rely on
FileNotFoundException to skip expired snapshots failed instead. A streaming
writer walking snapshots in latestSnapshotOfUser for its conflict-aware
writer clean checker could fail its checkpoint when a concurrent expiration
deleted the snapshot it was reading.

When reading fails with an IOException other than FileNotFoundException,
check whether the snapshot file still exists and report it as not found if
it is gone. The original failure is kept when the file exists or its
existence cannot be checked.
@sundapeng
sundapeng force-pushed the fix/snapshot-missing-on-read branch from b71cbd9 to f01ea7c Compare September 15, 2026 11:56
@sundapeng sundapeng changed the title [core] Report a snapshot deleted during reading as not found [core] Fix snapshot traversal failing when an old snapshot is expired concurrently Sep 15, 2026

@LsomeYeah LsomeYeah left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! I wonder whether it would be more appropriate to normalize this kind of missing-file exception at the FileIO layer?

SnapshotManager may not be the only caller that relies on FileNotFoundException for special handling. For example, incremental scans also distinguish a missing tag by catching FileNotFoundException before falling back to snapshot IDs. Perhaps handling this in FileIO could provide consistent behavior for snapshots, tags, and other metadata files.

@sundapeng sundapeng changed the title [core] Fix snapshot traversal failing when an old snapshot is expired concurrently [common] Fix snapshot and tag reads failing when the file is deleted concurrently Sep 15, 2026
@sundapeng

Copy link
Copy Markdown
Member Author

Thanks, that makes sense. Tag, Changelog and SchemaManager have the same gap, and all of them read through FileIO#readFileUtf8, so I moved the check there: when a read fails with an IOException other than FileNotFoundException and the file no longer exists, it throws FileNotFoundException. readOverwrittenFileUtf8 already does a similar exists() check for hint files.

I kept it in the default method instead of the OSS and Jindo FileIOs, because those can't tell a missing file from other failures either: hadoop-aliyun drops the OSS error code and fails with Null IO stream.

Move the existence check from SnapshotManager#tryFromPath into
FileIO#readFileUtf8, so tag, changelog and schema reads also treat a file
deleted during reading as not found.
@sundapeng
sundapeng force-pushed the fix/snapshot-missing-on-read branch from 1489001 to cae659f Compare September 15, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants