Conversation
7d6bcf7 to
b71cbd9
Compare
Some FileIOs report a snapshot that is deleted after its input stream was opened as a generic IOException on the first read, for example "404 Not Found / NoSuchKey" from an object store. SnapshotManager#tryFromPath turned that into a RuntimeException, so callers that rely on FileNotFoundException to skip expired snapshots failed instead. A streaming writer walking snapshots in latestSnapshotOfUser for its conflict-aware writer clean checker could fail its checkpoint when a concurrent expiration deleted the snapshot it was reading. When reading fails with an IOException other than FileNotFoundException, check whether the snapshot file still exists and report it as not found if it is gone. The original failure is kept when the file exists or its existence cannot be checked.
b71cbd9 to
f01ea7c
Compare
LsomeYeah
left a comment
There was a problem hiding this comment.
Thanks for the contribution! I wonder whether it would be more appropriate to normalize this kind of missing-file exception at the FileIO layer?
SnapshotManager may not be the only caller that relies on FileNotFoundException for special handling. For example, incremental scans also distinguish a missing tag by catching FileNotFoundException before falling back to snapshot IDs. Perhaps handling this in FileIO could provide consistent behavior for snapshots, tags, and other metadata files.
|
Thanks, that makes sense. I kept it in the default method instead of the OSS and Jindo FileIOs, because those can't tell a missing file from other failures either: hadoop-aliyun drops the OSS error code and fails with |
Move the existence check from SnapshotManager#tryFromPath into FileIO#readFileUtf8, so tag, changelog and schema reads also treat a file deleted during reading as not found.
1489001 to
cae659f
Compare
Purpose
Paimon treats a snapshot, tag, changelog or schema file as missing only when reading it throws
FileNotFoundException. Some FileIOs, such asOSSFileIOand Jindo, throw a plainIOExceptioninstead when the file is deleted while it is being read. A writer traversing snapshots inlatestSnapshotOfUserthen fails its checkpoint when it reaches a snapshot being expired, andTagManager#getfails instead of returning empty for a tag being deleted.When reading fails with another
IOException,FileIO#readFileUtf8now throwsFileNotFoundExceptionif the file is gone, and keeps the original error otherwise.Tests
FileIOTest: 2 new cases, 7 tests passed.SnapshotManagerTest: 1 new case, 43 tests passed.